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Preface  and  Acknowledgements 

Founded  in  2003,  the  Acquisition  Research  Program  (ARP)  has  become  an 
institutional,  multi-disciplinary  entity.  In  2006  the  ARP  made  significant  and  sustaining 
progress  toward  realizing  its  goals  to: 

1.  Position  NPS  as  a  recognized  leader  in  defense  acquisition  research. 

2.  Establish  NPS  acquisition  research  as  an  integral  part  of  policy-making  for 
Department  of  Defense  officials. 

3.  Create  a  stream  of  relevant  information  concerning  the  performance  of  DoD 
Acquisition  policies  with  viable  recommendations  for  continuous  process 
improvement. 

4.  Prepare  the  DoD  workforce  to  participate  in  the  continued  evolution  of  the 
defense  acquisition  process. 

5.  Collaborate  with  other  universities,  think  tanks,  industry  and  government  in 
acquisition  research. 

Since  inception,  over  100  reports  and  papers  have  been  published,  thereby  making  a 
significant  contribution  to  the  body  of  literature  on  the  defense  acquisition  process. 

Through  these  research  products,  ARP  sponsors  are  receiving  substantial  help  with  and 
insights  into  the  pressing  business  issues  of  the  day. 

The  synergy  between  faculty  research  and  student  classroom  instruction  has  been 
exceptional  with  many  relevant  and  current  instructional  materials  emerging  from  research 
products,  thus  enhancing  the  student  educational  experience.  Faculty  are  “refreshed”  in 
defense-relevant  subject  matter,  and  students  are  better  prepared  to  enter  the  acquisition 
work  force.  In  recognition  of  these  successes,  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  (USD  (AT&L))  provided  $1M  in  funding  for  additional 
projects.  This  funding  expands  the  ARP  by  30%  and  is  a  pilot  for  future  increases  in 
research  funding. 

Researcher  opportunities  provided  by  the  Chair  offer  significant  benefits  to 
researchers:  (1)  provision  of  funding  saving  researchers  “marketing"  time;  (2)  ties  with 
sponsor  POCs  thus  assuring  DOD  relevant  research;  (3)  assistance  with  final  formatting, 
editing  and  publishing  thus  relieving  researchers  from  the  “non-intellectual”  aspects  of  their 
research.  Each  of  these  is  a  substantial  benefit  but  the  growing  connectivity  between 
researchers  and  sponsors  is  paying  large  dividends  to  all  concerned.  While  we  at  the  Naval 
Postgraduate  School  like  to  think  of  our  institution  as  the  world’s  leader  in  defense 
acquisition  research,  we  also  recognize  that,  because  of  our  limited  size  and  resources,  we 
are  able  to  study  only  a  few  of  acquisition’s  myriad  of  complex  issues  and  challenges.  We 
know  that  genuine  progress  in  acquisition  research  can  be  achieved  and  sustained  only  to 
the  extent  that  scholars  from  a  broad  range  of  institutions  and  disciplines  are  engaged  to 
participate.  Once  this  “critical  mass”  of  researchers  is  formed,  we  may  anticipate  that 
acquisition  will  become  a  field  of  its  own,  with  perhaps  a  variety  of  acquisition  journals, 
acquisition  conferences,  and  university  courses  in  acquisition  management  and  policy. 

Such  intellectual  capacity,  we  may  hope,  will  before  long  prevail  against  acquisition’s 
perennial  and  often  pernicious  problems. 
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Accordingly,  the  year  2006  was  especially  significant  for  the  NPS  Acquisition 
Research  Program  in  taking  major  strides  toward  expanding  the  program’s  reach  in 
important  ways  to  other  institutions.  The  number  of  research  institutions  participating  as 
collaborators  grew  to  35  with  the  formation  of  a  Virtual  University  Consortium.  Most 
noteworthy  was,  as  mentioned  above,  our  securing  sponsorship  from  USD(AT&L)  to  fund 
research  proposals  selected  from  a  nationwide  call,  or  Broad  Agency  Announcement  (BAA) 
(copy  available  at  www.acquisitionresearch.ora).  We’re  truly  excited  at  the  prospects  of 
receiving  innovative  and  cutting  edge  proposals  from  the  top  minds  around  the  country.  We 
trust  that  this  new  sponsorship  will  act  like  good  seeds  sown  in  fertile  soil,  yielding  rich  fruits 
of  profitable  acquisition  research  for  many  years  to  come. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  Acquisition 
Research  Program: 

•  Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics) 

•  Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 

•  Program  Manager  (Infantry  Combat  Equipment) 

•  Program  Executive  Officer  (Integrated  Warfare  Systems) 

•  Program  Executive  Officer  (Littoral  and  Mine  Warfare) 

•  Project  Manager  (Modular  Brigade  Enhancements) 

•  Program  Executive  Officer  (Ships) 

•  Dean  of  Research,  Naval  Postgraduate  School 

We  also  thank  UGS  Corporation  and  the  Naval  Postgraduate  School  Foundation  and 
acknowledge  their  generous  contributions  in  support  of  this  symposium. 


James  B.  Greene,  Jr. 

Rear  Admiral,  US  Navy  (ret) 


Keith  F.  Snider,  PhD 
Associate  Professor 
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The  NPS  ‘‘A  Team” 


Rear  Admiral  James  B.  Greene,  Jr.  USN  (Ret.)  —  Acquisition  Chair,  Naval 
Postgraduate  School.  RADM  Greene  develops,  implements  and  oversees  the  Acquisition 
Research  Program  in  the  Graduate  School  of  Business  and  Public  Policy.  He  interfaces  with 
the  DoD,  industry  and  government  leaders  In  acquisition,  supervises  student  MBA  projects 
and  conducts  guest  lectures  and  seminars.  Before  serving  at  NPS,  RADM  Greene  was  an 
independent  consultant  focusing  on  Defense  Industry  business  development  strategy  and 
execution  (for  both  the  public  and  private  sectors),  minimizing  lifecycle  costs  through 
technology  applications,  alternative  financing  arrangements  for  capital-asset  procurement, 
and  “red-teaming”  corporate  proposals  for  major  government  procurements. 

RADM  Greene  served  as  the  Assistant  Deputy  Chief  of  Naval  Operations  (Logistics) 
in  the  Pentagon  from  1991-1995.  As  Assistant  Deputy,  he  provided  oversight,  direction  and 
budget  development  for  worldwide  US  Navy  logistics  operations.  He  facilitated  depot 
maintenance,  supply-chain  management,  base/station  management,  environmental 
programs  and  logistic  advice  and  support  to  the  Chief  of  Naval  Operations.  Some  of  his 
focuses  during  this  time  were  leading  Navy-wide  efforts  to  digitize  all  technical  data  (and, 
therefore,  reduce  cycle  time)  and  to  develop  and  implement  strategy  for  procurement  of 
eleven  Sealift  ships  for  the  rapid  deployment  forces.  He  also  served  as  the  Senior  Military 
Assistant  to  the  Under  Secretary  of  Defense  (Acquisition)  from  1987-1990  where  he  advised 
and  counseled  the  Under  Secretary  in  directing  the  DoD  procurement  process. 

From  1984-1987,  RADM  Greene  was  the  Project  Manager  for  the  Aegis  project.  This 
was  the  DoD’s  largest  acquisition  project  with  an  annual  budget  in  excess  of  $5  Billion/year. 
The  project  provided  oversight  and  management  of  research,  development,  design, 
production,  fleet  introduction  and  full  lifecycle  support  of  the  entire  fleet  of  Aegis  cruisers, 
destroyers  and  weapons  systems  through  more  than  2500  industry  contracts.  From  1980- 
1984,  RADM  Greene  served  as  Director,  Committee  Liaison,  Office  of  Legislative  Affairs 
followed  by  a  tour  as  the  Executive  Assistant,  to  the  Assistant  Secretary  of  the  Navy 
(Shipbuilding  and  Logistics).  From  1964-1980,  RADM  Greene  served  as  a  Surface  Warfare 
Officer  in  various  duties,  culminating  in  Command-at-Sea.  His  assignments  included 
numerous  wartime  deployments  to  Vietnam  as  well  as  the  Indian  Ocean  and  the  Persian 
Gulf. 


RADM  Greene  received  a  BS  in  Electrical  Engineering  from  Brown  University  in 
1964;  he  earned  a  MS  in  Electrical  Engineering  and  a  MS  in  Business  Administration  from 
the  Naval  Postgraduate  School  in  1973. 

Keith  F.  Snider  —  Associate  Professor  of  Public  Administration  and  Management  in 
the  Graduate  School  of  Business  &  Public  Policy  at  the  Naval  Postgraduate  School  in 
Monterey,  California,  where  he  teaches  courses  related  to  defense  acquisition  management. 
He  also  serves  as  Principal  Investigator  for  the  NPS  Acquisition  Research  Program  and  as 
Academic  Associate  for  resident  NPS  acquisition  curricula. 

Professor  Snider  has  a  PhD  in  Public  Administration  and  Public  Affairs  from  Virginia 
Polytechnic  Institute  and  State  University,  a  Master  of  Science  degree  in  Operations 
Research  from  the  Naval  Postgraduate  School,  and  a  Bachelor  of  Science  degree  from  the 
United  States  Military  Academy  at  West  Point.  He  served  as  a  field  artillery  officer  in  the  US 
Army  for  twenty  years,  retiring  at  the  rank  of  Lieutenant  Colonel.  He  is  a  former  member  of 
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the  Army  Acquisition  Corps  and  a  graduate  of  the  Program  Manager’s  Course  at  the 
Defense  Systems  Management  College. 

Professor  Snider’s  recent  publications  appear  in  American  Review  of  Pubiic 
Administration,  Administration  and  Society,  Administrative  Theory  &  Praxis,  Journai  of  Public 
Procurement,  Acquisition  Review  Quarterly,  and  Project  Management  Journal. 

Karey  L.  Shaffer  —  Program  Manager  for  the  Acquisition  Research  Program  at  the 
Graduate  School  of  Business  and  Public  Policy,  Naval  Postgraduate  School.  As  PM,  Shaffer 
is  responsible  for  operations  and  publications  in  conjunction  with  the  Acquisition  Chair  and 
the  Principal  Investigator.  She  has  also  catalyzed,  organized  and  managed  the  Acquisition 
Research  Symposiums  hosted  by  NPS. 

Shaffer  has  also  served  as  an  independent  Project  Manager  and  Marketing 
Consultant  on  various  projects.  Her  experiences  as  such  were  focused  on  creating 
marketing  materials,  initiating  web  development,  assembling  technical  teams,  managing 
project  lifecycles,  processes  and  cost-savings  strategies. 

From  2001-2002,  Shaffer  contracted  to  work  as  the  Executive  Assistant  to  the  Vice 
President  for  Leadership  and  Development  Human  Resources  for  Metris  Companies  in 
Minneapolis.  In  this  capacity,  she  introduced  project  lifecycle  and  process  improvements  to 
increase  efficiency.  Likewise,  as  a  Resource  Specialist  contractor  at  Watson  Wyatt 
Worldwide  in  Minneapolis,  she  developed  and  implemented  template  plans  to  address 
continuity  and  functionality  in  corporate  documents;  in  this  same  position,  she  introduced 
process  improvements  to  increase  efficiency  in  presentation  and  proposal  production  in 
order  to  reduce  the  instances  of  corruption  and  loss  of  vital  technical  information. 

Shaffer  has  also  served  as  the  Project  Manager  for  Imagicast,  Inc.  and  as  the 
Operations  Manager  for  the  Montana  World  Trade  Center.  At  Imagicast,  she  was  asked  to 
take  over  the  project  management  of  four  failing  pilots  for  Levi  Strauss  in  the  San  Francisco 
office.  Within  four  months,  the  pilots  were  released;  the  project  lifecycle  was  shortened;  and 
the  production  process  was  refined.  In  this  latter  capacity  at  the  MWTC,  Shaffer  developed 
operating  procedures,  policies  and  processes  in  compliance  with  state  and  federal  grant 
law.  Concurrently,  she  managed  $1 .25  million  in  federal  appropriations,  developed 
budgeting  systems  and  secured  a  $400,000  federal  technology  grant.  As  the  Operations 
Manager,  she  also  designed  MWTC’s  Conference  site,  managed  various  marketing 
conferences,  and  taught  student  practicum  programs  and  seminars. 

Shaffer  has  her  BA  in  Business  Administration  (focus  on  International  Business, 
Marketing  and  Management)  from  the  University  of  Montana.  She  is  currently  earning  her 
MBA  from  San  Francisco  State  University. 

A  special  thanks  to  our  editor  Jeri  Larsen  for  all  that  she  has  done  to  make  this 
publication  a  success,  to  David  Wood  and  Carl  Matsen  for  production,  to  Ian  White  for 
graphic  support,  to  Lindsay  D’Penha  for  CD  programming,  to  Jordy  Boom  for  conference 
website  development.  We  would  like  acknowledge  Arlene  Pulido,  Jennifer  Watson,  Bon 
Troung,  Toan  Tran  and  Jason  Munoz  of  the  staff  at  the  Graduate  School  of  Business  & 
Public  Policy  for  all  the  administrative  support  on  the  backend  to  make  the  Symposium  a 
success.  Our  program  success  is  directly  related  to  the  combined  efforts  of  many. 
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ACQUISITION  RESEARCH: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 
5**'  Annual  Acquisition  Research  Symposium 
May  14-15,  2008 
Monterey,  Caiifornia 


Announcement  and  Caii  for  Proposais 

The  Graduate  School  of  Business  &  Public  Policy  at  the  Naval  Postgraduate  School 
announces  the  5**^  Annual  Acquisition  Research  Symposium  to  be  held  May  14-15,  2008 
in  Monterey,  California. 

This  symposium  serves  as  a  forum  for  the  presentation  of  acquisition  research  and 
the  exchange  of  ideas  among  scholars  and  practitioners  of  public-sector  acquisition.  We 
seek  a  diverse  audience  of  influential  attendees  from  academe,  government,  and  industry 
who  are  well  placed  to  shape  and  promote  future  research  in  acquisition. 

The  Symposium  Program  Committee  solicits  proposals  for  panels  and/or  papers 
from  academicians,  practitioners,  students  and  others  with  interests  in  the  study  of 
acquisition.  The  following  list  of  topics  is  provided  to  indicate  the  range  of  potential  research 
areas  of  interest  for  this  symposium;  acquisition  and  procurement  poiicy,  supply  chain 
management,  public  budgeting  and  finance,  cost  management,  project  management, 
logistics  management,  engineering  management,  outsourcing,  performance 
measurement,  and  organization  studies. 

Proposals  must  be  submitted  by  November  9,  2007.  The  Program  Committee  will 
make  notifications  of  accepted  proposals  by  December  7,  2007.  Final  papers  must  be 
submitted  by  April  4,  2008  to  be  included  in  the  Symposium  Proceedings. 

Proposals  for  papers  should  include  an  abstract  along  with  identification,  affiliation, 
and  contact  information  for  the  author(s).  Proposals  for  papers  plan  for  a  20  minute 
presentation.  Proposals  for  panels  (plan  for  90  minute  duration)  should  include  the  same 
information  as  above  as  well  as  a  description  of  the  panel  subject  and  format,  along  with 
participants’  names,  qualifications  and  the  specific  contributions  each  participant  will  make 
to  the  panel. 

Submit  paper  and  panel  proposals  to  www.researchsvmposium.orq  . 
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NPS  BAA-07-002 

BROAD  AGENCY  ANNOUNCEMENT 
Acquisition  Research  Program 
Navai  Postgraduate  Schooi 

Open  untii  4:00  pm  POST  1  June  2007 

Primary  objective  is  to  attract  outstanding  researchers  and  scholars  to 
investigate  topics  of  interest  to  the  defense  acquisition  community.  The 

program  solicits  innovative  proposals  for  defense  acquisition  management  and 
policy  research  to  be  conducted  during  fiscal  year  (FY)  2008  (1  Oct  07  -30  Sep  08). 

Defense  acquisition  management  and  poiicy  research  refers  to 
investigations  in  all  disciplines,  fieids,  and  domains  that  (1)  are  involved  in  the 
acquisition  of  products  and/or  services  for  nationai  defense,  or  (2)  could 
potentially  be  brought  to  bear  to  improve  defense  acquisition.  It  includes  but  is 
not  limited  to  economics,  finance,  financial  management,  information  systems, 
organization  theory,  operations  management,  human  resources  management,  and 
marketing,  as  well  as  the  “traditional”  acquisition  areas  such  as  contracting, 
program/project  management,  logistics,  and  systems  engineering  management. 

This  program  is  targeted  in  particuiar  to  U.S.  universities  (including  U.S. 
government  schools  of  higher  education)  or  other  research  institutions 
outside  the  Department  of  Defense. 

The  Government  anticipates  making  multiple  awards  up  to  $100,000  each  for 
a  basic  research  period  of  twelve  months.  NPS  plans  to  complete  proposal 
evaluations  and  notify  awardees  in  early  August  2007. 


Full  Text  for  NPS  BAA-07-002 
at 

http://www.nps.edu/Research/WorkinaWithNPS.html 
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Keynote  Speaker 


Wednesday, 
May  16,  2007 

Keynote  Speaker 

8:00  a.m.  - 

Keynote  Speaker 

9:15  a.m. 

The  Honorable  Delores  M.  Etter  -  Assistant  Secretary  of  the  Navy  for 
Research,  Development  and  Acquisition 

The  Honorable  Delores  M.  Etter,  Assistant  Secretary  of  the  Navy 
for  Research,  Development  and  Acquisition,  was  nominated  on 
September  6,  2005  by  President  George  W.  Bush  to  serve  as  the 
Assistant  Secretary  of  the  Navy  for  Research,  Development  and 
Acquisition.  Dr.  Etter  was  then  sworn  in  on  November  7,  2005.  As 
the  Navy's  Senior  Acquisition  Executive,  Dr.  Etter  is  responsible  for 
research,  development,  and  acquisition  within  the  Department  of 
the  Navy.  From  August  2001  to  November  2005,  Dr.  Etter  was  a 
member  of  the  Electrical  Engineering  faculty  at  the  United  States 
Naval  Academy.  She  was  also  the  first  recipient  of  the  Office  of 
Naval  Research  Distinguished  Chair  in  Science  and  Technology. 

Her  academic  interests  were  in  digital  signal  processing  and 
communications.  Her  research  interests  included  biometric  signal 
processing,  with  an  emphasis  on  identification  using  iris  recognition. 

She  has  also  written  several  textbooks  on  computer  languages  and  software  engineering. 

From  June  1998  through  July  2001 ,  Dr.  Etter  served  as  the  Deputy  Under  Secretary  of  Defense  for 
Science  and  Technology.  In  that  position,  she  was  responsible  for  Defense  Science  and  Technology 
strategic  planning,  budget  allocation,  and  program  execution  and  evaluation  for  the  DoD  Science  and 
Technology  Program.  Dr.  Etter  was  the  Principal  U.S.  representative  to  the  NATO  Research  and 
Technology  Board.  She  was  also  responsible  for  the  Defense  Modeling  and  Simulation  Organization, 
the  High  Performance  Computing  Modernization  Office,  and  for  technical  oversight  of  the  Software 
Engineering  Institute.  Dr.  Etter  was  also  the  senior  civilian  in  charge  of  the  DoD  high-energy  laser 
research  program. 

From  1990-98,  Dr.  Etter  was  a  Professor  of  Electrical  and  Computer  Engineering  at  the  University  of 
Colorado,  Boulder.  During  1979-89,  Dr.  Etter  was  a  faculty  member  in  Electrical  and  Computer 
Engineering  at  the  University  of  New  Mexico.  She  served  as  Associate  Vice  President  for  Academic 
Affairs  in  1989.  During  the  1983-84  academic  year  she  was  a  National  Science  Foundation  Visiting 
Professor  in  the  Information  Systems  Laboratory  in  the  Electrical  Engineering  Department  at  Stanford 
University. 

Dr.  Etter  is  a  member  of  the  National  Academy  of  Engineering.  She  is  also  a  former  member  of  the 
National  Science  Board  and  the  Defense  Science  Board.  She  is  a  Fellow  of  the  Institute  of  Electrical 
and  Electronic  Engineers  (IEEE),  the  American  Association  for  the  Advancement  of  Science  (AAAS), 
and  the  American  Society  for  Engineering  Education  (ASEE).  She  served  as  President  of  the  IEEE 
Acoustics,  Speech,  and  Signal  Processing  Society  from  1988-89,  and  was  Editor-in-Chief  of  the  IEEE 
Transactions  on  Signal  Processing  from  1993-95. 

Dr.  Etter  was  a  member  of  the  Naval  Research  Advisory  Committee  from  1991-97,  and  chaired  the 
committee  from  1995-97.  She  has  received  the  Department  of  the  Navy  Distinguished  Public  Service 
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Award,  the  Secretary  of  Defense  Outstanding  Public  Service  Medal,  and  the  Department  of  Defense 
Distinguished  Public  Service  Medal. 
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Panel  1  -  Plenary  Panel  -  Acquisition  Reform 


Wednesday, 
May  16,  2007 


9:30  a.m.  -  Chair: 
11:00  a.m. 


Jacques  S.  Gansler,  Director,  Center  for  Public  Policy  &  Private 
Enterprise,  University  of  Maryland,  former  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics 

Discussants: 

Wiiliam  C.  Greenwait,  Deputy  Under  Secretary  of  Defense  for  Industrial 
Policy,  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology 
and  Logistics) 

Robert  H.  Trice,  Senior  Vice  President,  Business  Development, 

Lockheed  Martin  Corporation 

Pierre  Chao,  Senior  Fellow  and  Director  of  Defense  Industrial  Initiatives, 
Center  for  Strategic  and  International  Studies 


Chair:  Jacques  S.  Gansier,  former  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics,  is  the  first  holder  of  the  Roger  C.  Lipitz  Chair  in  Public  Policy  and  Private  Enterprise.  As  the 
third  ranking  civilian  at  the  Pentagon  from  1 997  to  2001 ,  Professor  Gansler  was  responsible  for  all 
research  and  development,  acquisition  reform,  logistics,  advanced  technology,  environmental 
security,  defense  industry,  and  numerous  other  security  programs.  Before  joining  the  Clinton 
Administration,  Dr.  Gansler  held  a  variety  of  positions  in  government  and  the  private  sector,  including 
Deputy  Assistant  Secretary  of  Defense  (Material  Acquisition),  Assistant  Director  of  Defense  Research 
and  Engineering  (Electronics),  Vice  President  of  ITT,  and  engineering  and  management  positions 
with  Singer  and  Raytheon  Corporations.  Throughout  his  career.  Dr.  Gansler  has  written,  published 
and  taught  on  subjects  related  to  his  work.  He  is  the  author  of  Defense  Conversion:  Transforming  the 
Arsenai  of  Democracy,  MIT  Press,  1995;  Affording  Defense,  MIT  Press,  1989,  and  The  Defense 
Industry,  MIT  Press,  1980.  He  has  published  numerous  articles  in  Foreign  Affairs,  Harvard  Business 
Review,  International  Security,  Public  Affairs,  and  other  journals  as  well  as  newspapers  and  frequent 
Congressional  testimonies.  He  is  a  member  of  the  National  Academy  of  Engineering  and  a  Fellow  of 
the  National  Academy  of  Public  Administration. 

Discussant:  William  C.  Greenwait,  Deputy  Under  Secretary  of  Defense  for  Industrial  Policy,  Office 
of  the  Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics),  Is  the  principal  advisor  to 
the  Under  Secretary  of  Defense  (Acquisition,  Technology  &  Logistics)  and  the  Deputy  Under 
Secretary  of  Defense  (Acquisition  and  Technology)  on  all  matters  relating  to  the  defense  industrial 
base.  His  office  is  responsible  for  ensuring  that  DoD  policies,  procedures,  and  actions  stimulate  and 
support  vigorous  competition  and  innovation  in  the  industrial  base  supporting  defense;  and  establish 
and  sustain  cost-effective  industrial  and  technological  capabilities  that  assure  military  readiness  and 
superiority. 

Prior  to  joining  DoD,  Mr  Greenwait  was  a  Professional  Staff  Member  of  the  Senate  Armed  Services 
Committee  (Senator  John  Warner,  Chairman)  from  March  1999  until  March  2006  and  was 
responsible  for  defense  acquisition  policy,  industrial  base,  export  control,  and  management  reform 
Issues.  In  addition,  from  January  2004,  he  served  as  deputy  to  the  staff  director  and  provided 
oversight  and  management  direction  of  the  committee’s  legislative  activities.  He  was  also  a  lead  staff 
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member  for  the  Subcommittee  on  Readiness  and  Management  Support  and  the  Subcommittee  on 
Strategic  Forces.  Previously,  he  served  on  the  Senate  Governmental  Affairs  Committee  (Senator 
Fred  Thompson,  Chairman)  as  a  Professional  Staff  Member  responsible  for  federal  management 
issues  and  committee  press  relations. 

Mr  Greenwalt  also  served  as  a  staff  member  for  the  Senate  Subcommittee  on  Oversight  of 
Government  Management  (Senator  William  Cohen,  Chairman)  where  he  was  responsible  for 
legislative  efforts  to  reform  federal  information  technology  acquisition  culminating  in  the  Clinger- 
Cohen  Act  of  1996.  Prior  to  coming  to  the  Senate  in  1994,  Mr  Greenwalt  was  a  visiting  fellow  at  the 
Center  for  Defense  Economics,  University  of  York,  England,  worked  for  the  Immigration  and 
Naturalization  Service  in  Frankfurt,  Germany  and  served  as  an  evaluator  with  the  US  General 
Accounting  Office  in  Los  Angeles,  California  where  he  specialized  in  defense  acquisition  issues. 

Mr  Greenwalt  graduated  from  California  State  University  at  Long  Beach  in  1982  with  a  degree  in 
political  science  and  economics  and  received  his  MA  in  defense  and  security  studies  from  the 
University  of  Southern  California  in  1989.  He  is  married  to  Paula  Mathews  and  they  live  with  their  son, 
Geoffrey,  and  daughter  Jenna,  in  Arlington,  VA. 

Discussant:  Pierre  Chao  is  a  Senior  Fellow  and  Director  of  Defense  Industrial  Initiatives,  Center  for 
Strategic  and  International  Studies.  Before  joining  CSIS  in  2003,  Pierre  was  a  managing  director  and 
senior  aerospace/defense  analyst  at  Credit  Suisse  First  Boston  from  1999-2003,  where  he  was 
responsible  for  following  the  U.S.  and  global  aerospace/defense  industry.  He  remained  a  CSFB 
independent  senior  adviser  from  2003-2006. 

Prior  to  joining  CSFB,  Pierre  was  the  senior  aerospace/defense  analyst  at  Morgan  Stanley  Dean 
Witter  from  1995-1999.  He  served  as  the  senior  aerospace/defense  industry  analyst  at  Smith  Barney 
during  1994  and  as  a  director  at  JSA  International,  a  Boston/Paris-based  management-consulting 
firm  that  focused  on  the  aerospace/defense  industry  (1986-88,  1990-93).  Pierre  was  also  a  co¬ 
founder  of  JSA  Research,  an  equity  research  boutique  specializing  in  the  aerospace/defense 
industry.  Before  signing  on  with  JSA,  he  worked  in  the  New  York  and  London  offices  of  Prudential- 
Bache  Capital  Funding  as  a  mergers  and  acquisitions  banker  focusing  on  aerospace/defense  (1988- 
90). 

Pierre  garnered  numerous  awards  while  working  on  Wall  Street.  Institutional  Investor  ranked  Pierre's 
team  the  number  one  global  aerospace/defense  group  every  year  eligible  from  2000-02  and  he  was 
on  the  Institutional  Investor  All-America  Research  Team  every  year  eligible  from  1996-2002.  He  was 
ranked  the  number  one  aerospace/defense  analyst  by  corporations  in  the  1998-2000  Reuters  Polls, 
the  number  one  aerospace/defense  analyst  in  the  1995-99  Greenwich  Associates  polls,  and 
appeared  on  the  Wall  Street  Journal  All-Star  list  in  four  of  seven  eligible  years. 

In  2000,  Pierre  was  appointed  to  the  Presidential  Commission  on  Offsets  in  International  Trade.  He 
was  a  member  of  the  2005  Defense  Science  Board  Summer  Study  (Assessment  of  Transformation), 
2006  DSB  Summer  Study  (Strategic  Technology  Vectors),  and  the  2006/2007  DSB  Task  Force  on 
the  Health  of  the  Defense  Industry.  He  is  also  a  guest  lecturer  at  the  National  Defense  University  and 
the  Defense  Acquisition  University.  Pierre  has  been  sought  out  as  an  expert  analyst  of  the  defense 
and  aerospace  industry  by  the  Senate  Armed  Services  Committee,  the  House  Science  Committee, 
Office  of  the  Secretary  of  Defense,  DoD  Defense  Science  Board,  Army  Science  Board,  NASA,  DGA 
(France),  NATO  and  the  Aerospace  Industries  Association  Board  of  Governors. 

Pierre  earned  dual  Bachelor  of  Science  degrees  in  Political  Science  and  Management  Science  from 
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Panel  2  -  Crossing  the  Valley  of  Death:  Technology 
Transition 


Wednesday, 
May  16,  2007 


11:15  a.m.-  Chair: 
12:45  p.m.  , 


John  J.  Kubricky,  Deputy  Under  Secretary  of  Defense  (Advance  Systems 
and  Concepts) 

Discussant: 

Alian  Shaffer,  Director,  Plans  Programs  and  Office,  Director  Defense 
Research  and  Engineering 


Papers: 


Stronger  Practices  Needed  to  Improve  DoD  Technology  Transition 
Practices 

Michael  Sullivan,  US  Government  Accountability  Office 

Crossing  the  Technology  Transfer  Chasm:  Network  Externalities, 
Coordination  Games  and  Lessons  Learned  from  ACTDs 

Peter  Coughlan,  Nicholas  Dew  and  William  (Bill)  Gates,  Naval 
Postgraduate  School 

Development  vs.  Deployment:  How  Mature  Should  a  Technology  Be 
before  It  Is  Considered  for  Inclusion  in  an  Acquisition  Program? 

Michael  J.  Pennock,  William  B.  Rouse  and  Diane  L.  Kollar,  Tennenbaum 
Institute,  Georgia  Institute  of  Technology 


Chair:  John  J.  Kubricky,  Deputy  Under  Secretary  of  Defense  (Advanced  Systems  and  Concepts), 
has  oversight  responsibilities  for  technology  transition  and  transfer  programs  to  include:  Advanced 
Concept  and  Joint  Capability  Technology  Demonstrations,  Joint  Warfighting  Program,  Foreign 
Comparative  Test,  Defense  Acquisition  Challenge,  Technology  Transition  Initiative,  ManTech, 
Defense  Production  Act  Title  III,  Dual  Use  S&T,  Independent  Research  and  Development,  and 
TechLink. 

Prior  to  taking  this  position  in  October,  2006,  Mr.  Kubricky  was  the  Department  of  Homeland 
Security’s  Director  of  Systems  Engineering  and  Development  with  the  Science  &  Technology 
Directorate.  Mr.  Kubricky’s  primary  responsibility  was  to  integrate  proven  technologies  into  systems 
for  demonstration,  operational  test  and  evaluation,  and  pre-production  prototypes,  which  adaptation 
and  deployment  of  military  technologies  for  homeland  security  applications.  He  had  also  served  as 
the  Homeland  Security  Advanced  Research  Projects  Agency’s  (HSARPA)  acting  director  since  July 
2005,  where  more  than  100  advanced  technology  programs  were  managed  by  the  agency  under 
contracts  with  the  private  sector.  HSARPA  initiatives  include  a  Small  Business  Innovative  Research 
(SBIR)  program.  Technology  Transfer,  and  Rapid  Technology  Application  Program,  each  of  which 
transitioned  new  security  technologies  to  operational  customers. 
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Before  his  service  in  the  Federal  government,  Mr.  Kubricky’s  career  in  industry  was  as  a  prime 
systems  integrator  of  tactical  intelligence  systems  on  airborne,  ground  and  marine  platforms.  He  is 
credited  with  designing,  integrating  and  fielding  the  US  Army’s  first  multi-sensor  aircraft;  for 
developing  real-time  multi-sensor  fusion  work-stations  that  became  a  baseline  for  today’s  tactical 
intelligence  processing  systems;  and  for  prototypes  of  Unmanned  and  Optionally  Piloted  Airborne 
Vehicles,  which  achieved  over  25,000  flight  hours  in  world-wide  deployments.  In  addition  to 
innovations  in  tactical  intelligence  programs,  Mr.  Kubricky  is  a  veteran  program  manager  of  strategic 
reconnaissance  development,  manufacturing  and  test/evaluation  programs. 

Mr.  Kubricky  holds  a  Bachelor  of  Science  degree  from  Johns  Hopkins  University,  where  he  also 
attended  the  graduate  program  for  Industrial  Management.  Mr.  Kubricky’s  service  with  the  US  Army 
includes  a  tour  with  the  Ninth  Infantry  Division  in  Vietnam  during  1968  and  1969.  He  and  his  wife  Joy, 
have  one  son,  and  live  in  Maryland. 

Discussant:  Allan  Shaffer,  Director,  Plans  Programs  and  Office,  Director  Defense  Research  and 
Engineering,  is  responsible  for  budget  and  planning  oversight  of  the  entire  Department  of  Defense 
Science  and  Technology  Program.  Mr.  Shaffer  earned  a  Bachelor  of  Science  Degree  in  Mathematics 
from  the  University  of  Vermont  in  1976,  and  was  commissioned  in  the  United  States  Air  Force.  He 
was  then  assigned  to  the  University  of  Utah,  where  he  earned  a  second  Bachelor  of  Science  Degree 
in  Meteorology.  He  also  earned  a  Master  of  Science  in  Meteorology  from  the  Naval  Postgraduate 
School  (with  distinction). 

He  then  had  a  number  of  assignments  in  weather,  intelligence,  and  science  and  technology 
management.  He  was  assigned  to  Mather  AFB,  Sacramento,  California,  as  the  Wing  Weather  Officer 
to  the  323d  Flying  Training  Wing  and  the  320th  Bombardment  Wing  (Heavy).  In  1984,  he  was 
assigned  to  the  Foreign  Technology  Division  (FTD),  where  he  was  an  intelligence  analyst  and  staff 
meteorologist,  assessing  the  technical  capabilities  of  the  Former  Soviet  Union,  as  well  as  the 
performance  of  Soviet  weapons  systems,  including  assessment  of  the  impact  of  the  atmosphere  on 
Soviet  military  laser  systems.  While  at  Wright-Patterson,  Mr.  Shaffer  had  an  extended  temporary  duty 
as  officer  in  charge.  Weather  Support  Force,  Palmerola  Air  Base,  Honduras.  He  was  also  named  Air 
Weather  Service  Junior  Officer  of  the  Year. 

From  Ohio,  he  became  Commander,  Detachment  2,  7th  Weather  Squadron,  Hanau,  Germany, 
providing  all  weather  support  to  the  3rd  Armored  Division,  then  Chief,  Plans  and  Programs  at  the  5th 
Weather  Wing,  Langley  Air  Force  Base,  Virginia.  In  this  position,  Mr.  Shaffer  was  responsible  for  all 
combat  support  plans  and  financial  matters  for  approximately  50  weather  units  assigned  to  Air 
Combat  Command  and  the  US  Army  Forces  Command.  Additionally,  he  planned  and  integrated 
technology  capabilities  from  defense  laboratories  into  operations.  From  Langely,  in  1993,  he  was 
assigned  to  the  Industrial  College  of  the  Armed  Forces,  Ft.  McNair,  D.  C.,  where  he  was  awarded  a 
Master  of  Science  in  National  Resource  Management. 

Next,  he  was  assigned  to  Hq  Air  Force,  with  assignments  as  Chief,  Plans  Division;  Chief  Interagency 
Affairs;  and  Chief,  Resources  Division,  USAF  Directorate  of  Weather.  In  these  positions,  he  planned 
and  assigned  resource  allocation  for  existing  and  future  Air  Force  systems.  In  1996,  he  was  assigned 
as  Specialist  for  Battlespace  Environments,  Defense  Research  and  Engineering.  In  this  position,  he 
was  responsible  for  oversight  of  the  DoD  technology  base  programs  in  Oceanography,  Terrain  and 
Topography,  Meteorology  and  Space  Physics.  In  1998,  he  became  the  Military  Assistant  to  the 
Deputy  Under  Secretary  of  Defense  (Science  and  Technology) 

In  1999,  he  took  his  last  military  position  as  Director,  Plans  and  Programs,  Air  Force  Weather 
Agency,  Offutt  AFB,  Nebraska.  Upon  retirement  from  active  military  duty  in  2000,  he  returned  to  the 
Pentagon  as  a  career  civil  servant  and  became  Director,  Multi-Disciplinary  Systems,  Office  of  the 
Deputy  Under  Secretary  of  Defense  for  Science  and  Technology.  He  assumed  his  current  position  in 
March  2001. 
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stronger  Practices  Needed  to  Improve  DoD  Technology 
Transition  Processes 


Presenter:  Michael  Sullivan  currently  serves  as  Director,  Acquisition  and  Sourcing 
Management,  at  the  U.S.  Government  Accountability  Office  (GAO).  This  group  has  responsibility 
for  examining  the  effectiveness  of  agency  acquisition  and  procurement  practices  in  meeting  their 
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What  GAO  Found 

Leading  commercial  companies  use  three  key  techniques  for  successfully 
developing  and  transitioning  technologies,  with  the  basic  premise  being  that 
technologies  must  be  mature  before  transitioning  to  the  product  line  side. 

■  Strategic  planning  at  the  corporate  level:  Strategic  planning  precedes  technology 
development  so  managers  can  gauge  market  needs,  identify  the  most  desirable 
technologies,  and  prioritize  resources. 

■  Gated  management  reviews:  A  rigorous  process  is  used  to  ensure  a  technology’s 
relevancy  and  feasibility  and  enlist  product  line  commitment  to  use  the  technologies 
once  the  labs  are  finished  maturing  them. 

■  Corroborating  tools:  To  secure  commitment,  technology  transition  agreements 
solidify  and  document  specific  cost,  schedule,  and  performance  metrics  labs  need  to 
meet  for  transition  to  occur.  Relationship  managers  address  transition  issues  within 
the  labs  and  product  line  teams  and  across  both  communities.  Meaningful  metrics 
gauge  project  progress  and  process  effectiveness. 

Not  only  does  DOD  lack  the  breadth  and  depth  of  these  techniques,  the 
department  routinely  accepts  high  levels  of  technology  risk  at  the  start  of  major  weapon 
acquisition  programs.  The  acquisition  community  works  with  technologies  before  they 
are  ready  to  be  transitioned  and  takes  on  responsibility  for  technology  development  and 
product  development  concurrently,  as  shown  in  the  following  figure.  A  defined  phase  for 
technology  transition  is  not  evident.  These  shortcomings  contribute  significantly  to 
DOD’s  poor  cost  and  schedule  outcomes. 
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Why  GAO  did  this  Study 

The  Department  of  Defense  (DOD)  relies  on  its  science  and  technology 
community  to  develop  innovative  technologies  for  weapon  systems,  spending  $13  billion 
on  basic,  applied,  and  advanced  technology  research.  Several  GAO  reports  have 
addressed  problems  in  transitioning  technologies  to  the  acquisition  community.  This 
report,  which  was  prepared  under  the  Comptroller  General’s  authority  to  conduct 
evaluations,  compares  DOD’s  technology  transition  processes  with  commercial  best 
practices.  Specifically,  GAO  identifies  technology  transition  techniques  used  by  leading 
companies  and  assesses  the  extent  to  which  DOD  uses  the  techniques. 

What  GAO  Recommends 

GAO  recommends  that  DOD  strengthen  its  technology  transition  processes  by 
developing  a  gated  process  with  criteria  to  support  funding  decisions;  expanding  the  use 
of  transition  agreements,  relationship  managers,  and  metrics;  and  setting  aside  funding 
for  transition  activities.  DOD  generally  agreed  with  GAO’s  recommendations  with  the 
exception  of  adopting  process-oriented  metrics  and  setting  aside  funding  for  transition.  It 
cited  ongoing  initiatives  it  believes  address  several  of  the  recommendations.  GAO 
believes  DOD’s  actions  to  date  are  incomplete  and  all  recommendations  warrant  further 
attention. 

www.gao.gov/cgi-bin/getrpt7GAO-06-883. 

To  view  the  full  product,  including  the  scope  and  methodology,  click  on  the  link  above. 
For  more  information,  contact  Michael  J.  Sullivan  at  (202)  512-4841  or 
sullivanm@gao.gov. 
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Abstract 

Technology  transfer,  getting  new  and  improved  weapon  systems  into  the  hands  of 
our  warfighters,  has  been  a  persistent  problem  in  the  Department  of  Defense.  Advanced 
Concept  Technology  Demonstrations  (ACTDs),  more  recently  re-designated  Joint  Concept 
Technology  Demonstrations  (JCTDs),  have  been  introduced  to  help  facilitate  the  technology 
transfer  process.  ACTDs  programs  are  designed  to  demonstrate  commercial-off-the-shelf 
(COTS)  technologies  that  can  be  modified  to  serve  joint  service  requirements.  COTS 
technology  and  joint  service  requirements  are  two  essential  features  for  ACTDs.  Once  the 
technology  has  been  demonstrated,  the  ACTD  is  expected  to  be  transferred  to  a  program  of 
record  for  acquisition  and  fielding. 

Unfortunately,  ACTDs  have  frequently  experienced  trouble  crossing  what  has  been 
referred  to  as  the  “chasm,”  the  transition  from  successful  technology  demonstration  and 
usage  by  early  adopters  to  acquisition  and  fielding  by  a  significant  share  of  potential  users. 

In  actuality,  the  transition  from  technology  demonstration  to  acquisition  and  fielding  for  joint 
service  technologies  involves  coordination  across  several  stakeholders  and  network 
externalities  (network  externalities  are  present  when  adoption  by  one  stakeholder  affects  the 
benefits  of  adoption  by  other  stakeholders — a  characteristic  common  in  joint  service 
programs). 

Analysis  of  these  programs  inevitably  raises  questions  about  the  causes  of  relative 
“success”  or  “failure”  of  technology  transfer,  particularly  when  the  programs  involve  a  large 
number  of  DoD  stakeholders.  Much  research  on  the  success  of  acquisition  programs 
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focuses  on  program  management.  For  example,  a  recent  GAO  report  criticized  the 
management  of  the  DoD’s  passive  RFID  acquisition  program  (Solis,  2005,  September).  We 
think  that  experimenting  with  coordination  (weak-link)  games  and/or  network  externality 
models  in  the  lab  might  shed  some  light  on  why  certain  types  of  coordination  problems  arise 
in  the  first  place,  how  decision-makers  can  be  expected  to  respond,  and  what  might  be  done 
about  them. 

Research  has  long  explored  these  issues.  The  key  insight  this  research  brings  to  the 
table  is  that  all  coordination  problems  generally  exhibit  the  same  underlying  properties — i.e., 
they  can  be  reduced  to  a  single  simple  and  parsimonious  “game”  called  a  “weak-link”  game 
that  is  played  by  multiple  individuals  who  depend  on  each  other  for  the  overall  result  or 
outcome  of  the  game.  This  weak-link  game  has  already  been  “played”  in  experimental 
economics  laboratories,  and  there  is  emerging  research  literature  on  the  properties  of  this 
game  and  the  implications  of  these  lab  discoveries  for  real-life  situations  that  are  analogous 
to  the  weak-link  game.  In  part,  this  research  plans  to  extend  this  model  to  coordination  in 
technology  transfer. 

At  the  same  time,  the  theory  of  network  externalities  is  well  developed;  but,  there  is 
limited  experimental  data  indicating  how  decision-makers  might  respond  to  alternative  policy 
mechanisms.  In  theory,  decision-makers  will  under-value  their  decisions  to  adopt  new 
technology  because  they  will  not  internalize  the  value  their  adoption  decisions  have  on  other 
potential  adopters.  This  might  help  explain  the  reluctance  of  individual  services  to  be  early 
adopters  for  joint  service  technologies.  Empirically  exploring  network  externalities  in 
experimental  settings  might  help  shed  light  on  how  to  best  address  this  market  imperfection 
in  military  technology  diffusion,  particularly  as  ACTDs  face  the  technology  transfer  chasm. 

This  research  examines  the  past  transition  successes  and  failures  of  ACTD 
programs  and  uses  this  experience  to  develop  a  model(s)  of  the  coordination  issues  and/or 
network  externalities  involved  in  the  technology  transfer  process.  Once  developed,  this 
research  proposes  to  develop  economic  experiments  involving  coordination  games  and 
network  externalities  to  see  if  this  empirical  evidence  can  emulate  the  technology  transfer 
chasm. 
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Abstract 

Modern  military  systems  increasingly  rely  on  the  integration  of  multiple  advanced 
technologies.  While  these  technologies  vastly  increase  warfighter  capabilities,  they  also 
introduce  risk  into  the  system  design  and  development  process  that  tends  to  increase  both 
its  cost  and  duration.  As  acquisition  cycle-times  increase,  warfighters  must  make  do  with 
dated  technology  for  longer  periods.  Thus,  there  is  an  incentive  to  push  as  many  advanced 
technologies  as  possible  into  each  program  to  maximize  warfighter  capability  over  the  next 
acquisition  cycle.  Unfortunately,  the  more  new  technologies  a  system  has,  the  more  risky  its 
acquisition  becomes,  and  consequently,  its  duration  and  cost  increase  even  further.  Thus, 
there  is  a  feedback  effect  that  exacerbates  the  problem.  Open-architecture  designs  can 
partially  alleviate  this  problem,  but  some  technology  decisions  are  so  integral  to  a  system’s 
design  that  they  cannot  be  relegated  to  future  upgrades.  Consequently,  there  is  a  tradeoff 
between  incorporating  these  technologies  now  and  increasing  program  risk  or  developing 
and  evaluating  them  further  but  potentially  postponing  their  application  to  future  acquisition 
cycles.  Our  paper  will  examine  this  tradeoff  by  considering  a  new  technology’s  contribution 
to  program  risk. 

Introduction 

Despite  repeated  attempts  at  reforming  the  defense  acquisition  process.  Defense 
Department  programs  continue  to  experience  substantial  cost  overruns,  schedule  delays, 
and  performance  shortfalls.  While  there  are  likely  multiple  causes  for  reform  failure,  this 
paper  aims  to  address  only  one  of  the  critical  issues  that  contribute  to  these  acquisition 
challenges.  That  issue  is  the  maturity  of  critical  technologies  employed  in  major  defense 
acquisition  programs. 

There  have  been  repeated  calls  for  the  Department  of  Defense  to  use  evolutionary 
rather  than  revolutionary  acquisition  strategies.  In  fact,  the  DoD  has  revised  its  acquisition 
polices  to  that  end  (GAO,  2003).  Despite  these  new  policies,  recent  GAO  reports  have 
indicated  that  most  major  acquisition  programs  are  still  revolutionary  rather  than  evolutionary 
and  do  not  follow  current  DoD  guidelines  for  knowledge-based  acquisition  (GAO,  2006,  April 
5;  2006,  April  13;  2006,  December  21).  It  seems  that  every  program  is  an  exception.  Why  is 
this? 


To  that  end,  this  paper  investigates  two  key  questions:  What  level  of  maturity  is 
acceptable  for  a  technology  to  be  included  in  a  major  acquisition  program,  and  what 
obstacles  prevent  the  DoD  from  implementing  an  evolutionary  acquisition  process? 

Our  findings  will  show  that,  relatively  speaking,  it  is  better  to  employ  mature 
technologies:  thus,  an  evolutionary  strategy  is  superior  under  most  circumstances  to  a 
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revolutionary  strategy  in  terms  of  getting  capabilities  delivered  to  the  warfighter.  We  also 
found,  however,  that  when  a  program  relies  on  multiple,  critical  technologies,  especially 
those  intended  for  a  multi-mission  role,  the  evolutionary  strategy  is  unstable.  There  is  a 
natural  tendency  to  revert  to  the  revolutionary  technology  strategy  even  though  it  is  not  in 
the  best  interest  of  the  warfighter. 

This  paper  is  structured  in  the  following  manner.  First,  we  discuss  the  background  of 
knowledge-based,  evolutionary  acquisition  and  why  it  is  considered  important  for  defense 
acquisition.  Second,  we  develop  a  high-level  simulation  model  of  acquisition  to  help  us 
investigate  these  issues.  Third,  we  use  the  model  to  analyze  defense  acquisition  policy 
alternatives  regarding  technological  maturity.  Finally,  we  conclude  with  the  policy 
implications  of  this  analysis. 

Background 

The  troubled  history  of  the  DoD  acquisition  system  (as  well  as  the  repeated  attempts 
to  reform  it)  are  well  known,  and  we  will  not  recount  them  here  (See  Pennock,  Rouse  & 
Kollar,  2007  and  GAO,  2006,  April  13).  Instead,  our  focus  will  be  on  the  more  recent 
attempts  to  reform  the  acquisition  system  by  employing  knowledge-based  business 
practices  and  evolutionary  acquisition. 

A  common  criticism  of  the  defense  acquisition  process  is  that  it  tends  to  emphasize 
large  leaps  in  capability  achieved  by  utilizing  promising  but  immature  technology.  Changes 
to  defense  acquisition  policy  over  the  last  several  years  have  attempted  to  reverse  this  trend 
by  creating  a  milestone  process  in  which  programs  must  meet  certain  requirements  before 
proceeding  from  one  phase  to  the  next  (DOD,  2003a,  2003b).  (See  Figure  1 .)  Part  of  this 
milestone  process  Is  an  assessment  of  the  maturity  of  technologies  to  be  employed  in 
acquisition  programs  as  well  as  a  plan  to  manage  their  development. 
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Figure  1.  Defense  Acquisition  Management  Framework 

(DoD,  2003b) 


Technological  maturity  is  typically  assessed  using  the  Technology  Readiness  Level 
(TRL)  scale  (Table  1 ).  The  TRL  scale  is  a  qualitative  assessment  scale  that  is  designed  to 
aid  decision-makers  by  providing  some  sense  of  a  given  technology’s  level  of  risk.  In 
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general,  one  assumes  that  the  higher  the  TRL  level,  the  less  uncertainty  a  technology  brings 
to  a  program.  It  is  important  to  note  that  the  TRL  scale  evaluates  a  technology  in  isolation 
and  does  not  consider  the  integration  risks  (Smailing  &  deWeck  2007).  Regardless,  the 
aforementioned  poiicy  changes  encourage  programs  to  utiiize  more  mature,  demonstrated 
technologies  (i.e.,  higher  TRL  ieveis)  rather  than  more  immature  and,  consequently,  more 
risky  technologies.  For  exampie,  quaiification  to  enter  the  system  deveiopment  phase 
nominaiiy  requires  aii  critical  technologies  to  be  at  TRL  level  6  or  higher  (though  the  GAO 
recommends  at  least  TRL  level  7  (GAO  2006,  April  13)). 


Technology  Readiness  Level 

Description 

1 .  Basic  principles  observed 
and  reported. 

Lowest  level  of  technology  readiness.  Scientific  research 
begins  to  be  translated  Into  applied  research  and 
development.  Examples  might  Include  paper  studies  of  a 
technology's  basic  properties. 

2.  Technology  concept  and/or 
application  formulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical 
applications  can  be  invented.  Applications  are  speculative,  and 
there  may  be  no  proof  or  detailed  analysis  to  support  the 
assumptions.  Examples  are  limited  to  analytic  studies. 

3.  Analytical  and  experimental 
critical  function  and/or 
characteristic  proof  of  concept. 

Active  research  and  development  is  initiated.  This  includes 
analytical  studies  and  laboratory  studies  to  physically  validate 
analytical  predictions  of  separate  elements  of  the 
technology.  Examples  include  components  that  are  not  yet 
integrated  or  representative. 

4.  Component  and/or 
breadboard  validation  in 
laboratory  environment. 

Basic  technological  components  are  integrated  to  establish  that 
they  will  work  together.  This  is  relatively  "low  fidelity"  compared 
to  the  eventual  system.  Examples  include  integration  of  "ad 
hoc"  hardware  in  the  laboratory. 

5.  Component  and/or 
breadboard  validation  in 
relevant  environment. 

Fidelity  of  breadboard  technology  increases  significantly.  The 
basic  technological  components  are  integrated  with  reasonably 
realistic  supporting  elements  so  it  can  be  tested  in  a  simulated 
environment.  Examples  include  "high  fidelity"  laboratory 
integration  of  components. 

6.  System/subsystem  model 
or  prototype  demonstration  in  a 
relevant  environment. 

Representative  model  or  prototype  system,  which  is  well 
beyond  that  of  TRL  5,  is  tested  in  a  relevant 
environment.  Represents  a  major  step  up  in  a  technology's 
demonstrated  readiness.  Examples  include  testing  a  prototype 
in  a  high-fidelity  laboratory  environment  or  in  simulated 
operational  environment. 

7.  System  prototype 
demonstration  in  an 
operational  environment. 

Prototype  near,  or  at,  planned  operational  system.  Represents 
a  major  step  up  from  TRL  6,  requiring  demonstration  of  an 
actual  system  prototype  in  an  operational  environment  such  as 
an  aircraft,  vehicle,  or  space.  Examples  Include  testing  the 
prototype  in  a  test-bed  aircraft. 

8.  Actual  system  completed 
and  qualified  through  test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under 
expected  conditions.  In  almost  all  cases,  this  TRL  represents 
the  end  of  true  system  development.  Examples  include 
developmental  test  and  evaluation  of  the  system  in  its  intended 
weapon  system  to  determine  if  it  meets  design  specifications. 

9.  Actual  system  proven 
through  successful  mission 
operations. 

Actual  application  of  the  technology  in  its  final  form  and  under 
mission  conditions,  such  as  those  encountered  in  operational 
test  and  evaluation.  Examples  include  using  the  system  under 
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operational  mission  conditions. 

Table  1.  DoD  Technology  Readiness  Levels 

(DoD,  2006,  Ch.  10.5.2) 


What  is  the  rationale  behind  a  policy  that  requires  a  relatively  mature  level  of 
technology?  The  issue  is  that  development  of  immature  technology  is  fairly  unpredictable 
in  terms  of  cost,  schedule,  and  efficacy.  When  a  program  contains  multiple  immature 
technologies,  these  tend  to  delay  the  program  and  add  cost.  If  technology  development  is 
done  in  concurrence  with  system  development,  the  problem  can  be  exacerbated  because 
unforeseen  outcomes  can  lead  to  significant  rework.  The  net  result  is  that,  on  average, 
programs  with  immature  technologies  will  take  longer  and  cost  more.  Consequently, 
warfighters  must  make  due  with  obsolete  equipment  longer,  thus  increasing  the  chances 
that  they  will  engage  in  combat  operations  with  less  capability  than  they  could  have  had 
otherwise. 

As  a  result,  it  would  seem  that  a  superior  approach  would  be  to  reduce  cycle-time  by 
setting  more  modest  goals  for  each  deployed  increment  of  capability.  This  is  often  referred 
to  as  an  evolutionary  rather  than  a  revolutionary  acquisition  process,  and  there  are  several 
ways  to  achieve  such  a  process.  First,  one  can  make  use  of  open-architecture  design  and 
spiral  development.  The  idea  behind  spiral  development  is  that  the  system  can  be  deployed 
with  an  initial  mature  technology,  which  can  then  be  upgraded  over  time  (Johnson  & 
Johnson,  2002).  This  approach  can  work  well  for  technologies  that  are  loosely  coupled  to 
the  system  design.  In  other  words,  there  is  a  clear,  well-defined  interface  such  that  changes 
in  the  implementation  of  the  subsystem  or  technology  to  be  upgraded  do  not  interfere  with 
the  rest  of  the  system.  Open  architecture  design  is  perfect  for  a  technology  such  as  a 
software  algorithm.  Assuming  that  the  software  interface  has  been  standardized,  it  is 
comparatively  straightforward  to  replace  an  old  software  component  with  a  new  one.  This 
approach,  in  fact,  has  been  demonstrated  successfully  on  submarine  acoustic  systems 
(Boudreau,  2006). 

When  technologies  or  subsystems  are  tightly  coupled  to  the  overall  system, 
however,  any  changes  to  the  design  of  the  subsystem  impact  the  design  of  the  whole 
system.  Thus,  open-architecture  design  is  not  always  a  feasible  alternative.  An  extreme 
example  would  be  the  hull-form  of  a  surface  combatant.  Take,  for  instance,  the  tumblehome 
hull  design  of  the  new  Zumwalt-class  destroyer.  If  some  critical  issues  were  to  arise  with  the 
hull  design,  it  is  likely  that  a  significant  portion  of  the  ship  would  have  to  be  redesigned.  Of 
course,  hull  form  is  a  rather  obvious  case,  but  there  are  many  mission-critical  systems  in  any 
modern  military  system  that  exhibit  varying  degrees  of  interaction  with  the  rest  of  the  system 
design.  Since  changes  to  these  systems  would  require  substantial  rework,  it  is  imperative 
that  they  be  mature  prior  to  system  integration,  hence  the  appeal  of  evolutionary  acquisition. 

Under  evolutionary  acquisition,  system  acquisition  cycles  are  more  rapid  and  make 
use  of  mature,  available  technology.  The  development  of  new  technologies  is  detached 
from  the  acquisition  process,  so  that  the  fate  of  a  program  does  not  hinge  on  the  success  or 
failure  of  any  one  risky  technology.  The  evolutionary  design  process  is  enforced  via  a 
knowledge-based  acquisition  process.  The  program  contains  a  number  of  evaluation  points 
or  milestones.  At  each  milestone,  the  program  must  demonstrate  that  it  has  met  certain 
developmental  requirements  in  order  to  proceed  to  the  next  phase.  For  example.  Milestone 
A  entails  requirements  such  as  an  Initial  Capabilities  Document,  an  Analysis  of  Alternatives 
(AoA),  a  Systems  Engineering  Plan  (SEP),  and  Technology  Readiness  Assessment. 
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Despite  the  fact  that  the  DoD  acknowledges  evolutionary  and  knowledge-based 
acquisition  as  best  practices  and  has  committed  them  to  policy,  recent  GAO  reports  have 
indicated  that  most  major  acquisition  programs  do  not  follow  these  polices  (GAO,  2006,  April 
5;  2006,  April  13;  2006,  December  21).  Consequently,  these  major  acquisition  programs 
have  continued  to  experience  significant  cost  overruns  and  major  delays.  In  particular, 
these  reports  have  indicated  that  most  major  acquisition  programs  are  revolutionary  rather 
than  evolutionary,  and  they  are  permitted  to  bypass  major  milestone  requirements.  Most 
rely  on  multiple  immature  technologies  that  are  not  fully  developed  before  overall  system 
development  begins.  The  Office  of  the  Secretary  of  Defense  (OSD)  has  acknowledged  that 
this  is  a  common  practice  (GAO  2006,  April  13). 

One  example  in  particular  that  makes  the  consequences  of  this  acquisition  approach 
clear  is  the  case  of  WIN-T  and  JNN-N.  The  Warfighter  Information  Network-Tactical  (WIN- 
T)  Is  the  next  generation  tactical  communications  network  for  the  US  Army  and  will  provide  a 
major  leap  forward  in  battlefield  communications.  However,  when  the  program  moved  into 
the  system-development  phase,  9  of  the  system’s  12  critical  technologies  were  immature 
(GAO  2006,  December  21).  As  a  result,  WIN-T  has  been  unavailable  for  both  Operation 
Iraqi  Freedom  and  Operation  Enduring  Freedom.  Because  it  was  determined  that  there  was 
an  urgent  need  for  better  battlefield  communications  to  support  these  two  operations,  the 
Joint  Network  Node-Network  (JNN-N)  program  was  created.  To  address  this  urgent  need, 
the  JNN-N  program  bypassed  many  of  the  normal  acquisition  procedures  to  accelerate 
fielding  of  the  system.  While  this  may  be  understandable  given  the  urgency  of  the  situation, 
acquisition  procedures  are  in  place  to  ensure  that  acquired  systems  function  properly  and 
are  cost-effective.  As  the  GAO  points  out: 

When  the  Army  opted  to  pursue  large  technology  advances  in  networking 
capabilities  to  support  the  future  forces  through  WIN-T,  rather  than  pursuing  a 
more  incremental  approach.  It  accepted  a  gap  in  providing  tactical  networking 
capabilities  to  the  warfighter  [...]  If  the  Army  had  followed  DOD’s  acquisition 
policy  preferences,  which  emphasize  achieving  capabilities  in  increments 
based  on  mature  technologies  to  get  capabilities  into  the  hands  of  the  user 
more  quickly,  it  might  have  been  able  to  get  needed  communications 
capabilities  to  the  warfighter  sooner.  (GAO  2006,  December  21 ) 

Thus,  a  more  evolutionary  approach  to  acquisition  may  have  reduced  the  risks  to  the 
warfighter  by  both  avoiding  capability  gaps  as  well  as  mitigating  the  need  for  emergency 
programs  that  bypass  the  usual  acquisition  procedures. 

To  summarize,  the  Department  of  Defense  claims  to  favor  evolutionary  acquisition, 
but  does  not  follow  through  in  practice.  The  GAO  asserts  that  there  are  a  number  of  causes 
for  this,  one  of  which  is  the  lack  of  mandatory  controls  on  the  milestone  process  (GAO  2003, 
2006,  April  5,  2006,  April  1 3,  2006,  December  21 ).  But  if  evolutionary  acquisition  is 
superior,  why  would  the  DoD  not  follow  its  tenets  even  without  the  mandatory  controls? 
There  are  really  two  possibilities.  Either  evolutionary  acquisition  is  not  the  best  approach 
and  when  given  the  flexibility  program  managers  avoid  it,  or  the  nature  of  the  acquisition 
system  itself  works  against  the  successful  implementation  of  evolutionary  methods. 
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Analysis  Approach 


To  better  understand  the  nature  of  evolutionary  acquisition,  we  must  model  the 
impact  of  a  program’s  technology  strategy  on  the  level  of  capability  actually  deployed  in  the 
field.  In  particular,  a  technology  strategy  consists  of  the  technologies  selected  to  improve 
each  capability  that  a  system  provides.  A  technology  policy  that  emphasizes  major 
increases  in  capability  would  likely  rely  on  immature  technology  and,  thus,  would  be  a 
revolutionary  strategy.  Consequently,  the  acquisition  program  will  require  a  substantial 
technology  development  phase.  On  the  other  hand,  a  technology  strategy  that  emphasizes 
small  improvements  in  technology  would  rely  on  more  mature  technology  and  could  be 
considered  an  evolutionary  strategy.  This  type  of  strategy  effectively  detaches  technology 
development  from  the  acquisition  program  and,  consequently,  would  have  a  relatively  short 
technology-development  phase. 

What  we  would  like  to  examine  is  the  impact  of  the  selected  technology  strategy  over 
the  long-term.  Thus,  we  are  concerned  with  the  deployed  capability  resulting  from  a 
sequence  of  acquisition  programs.  In  particular,  we  are  assuming  that  our  objective  is  to 
improve  the  capabilities  of  a  particular  class  of  system  such  as  a  surface  combatant  or  air 
superiority  fighter.  To  model  this,  we  must  establish  a  means  to  link  the  selected  technology 
strategy  to  the  time  required  to  complete  an  acquisition  program.  This  will  determine  when  a 
capability  improvement  is  deployed.  After  an  acquisition  program  completes,  we  assume 
that  another  begins  immediately  to  procure  the  next  iteration  of  that  system. 

To  accomplish  this,  we  will  assume  that  we  can  model  each  acquisition  program  as  a 
small  PERT  chart.  PERT  charts  are  a  common  program  management  tool  for  managing 
schedule  risk.  For  our  particular  model,  we  will  assume  a  fairly  simple  formulation.  We  will 
assume  that  there  is  a  technology-development  stage  followed  by  a  system-integration 
stage.  Each  acquisition  program  contains  a  number  of  critical  technologies  that  must  be 
developed  for  the  program  to  reach  a  successful  conclusion.  We  will  assume  that  each 
critical  technology  can  be  developed  in  parallel,  but  all  must  be  complete  before  system 
integration  can  begin.  This  is  an  admitted  simplification  that  works  both  for  and  against  the 
acquisition  program.  The  assumption  of  parallel  technology  development  is  somewhat 
optimistic  as  the  outcome  of  each  critical  technology  may  be  somewhat  interdependent. 

The  assumption  that  all  development  must  be  completed  is  somewhat  pessimistic  because 
some  integration  work  can  be  done  based  on  the  estimated  outcome  of  technology 
development.  However,  since  unanticipated  outcomes  in  the  technology-development 
phase  can  lead  to  substantial  rework  in  the  integration  phase,  this  is  not  an  unreasonable 
assumption.  Given  those  assumptions,  we  can  structure  each  acquisition  program  as 
shown  in  Figure  2. 
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Figure  2.  Simplified  PERT  Chart  for  an  Acquisition  Program 

Keeping  in  line  with  the  standard  PERT  formulation,  we  will  assume  that  the  duration 
of  each  technology  development  activity  is  stochastic  and  beta  distributed  (Figure  3).  The 
beta  distribution  is  appealing  is  this  context  because  it  has  finite  upper  and  lower  bounds  on 
the  activity  duration,  hence  its  use  in  PERT. 


Probability  of  Activity  Duration 


Figure  3.  Example  Beta  Distribution  (a  =  2,  p  =  5) 


One  notion  we  would  like  to  capture  is  the  relationship  between  the  maturity  of  a 
technology  selected  for  an  acquisition  program  and  the  amount  of  schedule  risk  it  entails.  It 
is  fairly  safe  to  assume  that  the  more  immature  a  technology  is,  the  more  schedule  risk  there 
is  in  its  development.  In  fact,  we  can  go  one  step  further  and  assume  that  it  follows  the  law 
of  diminishing  returns.  In  other  words,  each  additional  increment  of  schedule  risk  that  we 
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accept  buys  a  reduced  amount  of  gain  in  capability.  To  make  this  relationship  more 
concrete,  we  must  select  metrics  for  the  gain  in  capability  and  the  level  of  schedule  risk.  For 
the  former,  we  will  consider  the  percent  gain  in  capability  over  the  currently  deployed 
capability.  Thus,  a  relatively  low  percent  gain  would  be  considered  an  evolutionary 
technology  whereas  a  large  percent  gain  would  be  a  revolutionary  technology.  Since  we 
would  only  accept  an  immature  technology  in  exchange  for  an  increase  in  capability,  we  can 
assume  that  for  the  purposes  of  our  model,  the  percent  gain  in  capability  is  also  an 
acceptable  proxy  for  technological  maturity.  As  for  the  risk,  we  will  assume  that  schedule 
risk  is  encapsulated  in  the  upper  bound  of  the  probability  distribution  for  the  duration  of 
technology  development.  For  the  sake  of  simplicity,  the  lower  bound  and  shape  parameters 
of  the  beta  distribution  will  remain  constant.  Thus,  if  we  select  a  particular  percent  gain  in 
capability  as  our  technology  policy,  it  determines  a  particular  upper  bound  on  the  distribution 
of  the  development  time  of  that  technology.  This  is  illustrated  in  Figure  4.  When  we  change 
the  upper  bound  of  the  distribution,  two  things  occur.  We  increase  the  expected  time  to 
develop  the  technology,  and  we  increase  the  spread  of  the  distribution. 


Risk  -  Return  Tradeoff 


Figure  4.  Tradeoff  between  Risk  (the  Upper  Bound  on  the  Duration  of  Technology 
Development)  and  Return  (the  Growth  in  Capability) 


We  define  a  technology  policy  as  the  targeted  percent  gain  in  capability  for  each 
acquisition  program.  Thus,  if  a  more  aggressive  target  is  selected,  there  will  be  a  greater 
increase  in  capability  for  each  new  system  deployed.  However,  the  expected  duration  of  the 
acquisition  cycle  will  also  increase.  Given  our  model  structure,  we  can  use  Monte  Carlo 
simulation  to  generate  possible  capability  trajectories.  This  is  accomplished  in  the  following 
manner.  First,  sample  from  the  beta  distribution  for  each  technology  is  included  in  the 
acquisition  program.  The  integration  phase  cannot  begin  until  all  technology  development  is 
complete,  so  the  longest  sampled  time  dictates  the  length  of  the  technology-development 
phase.  That  time  plus  the  time  required  for  integration  is  the  total  time  required  for  the 
acquisition  program.  At  the  end  of  the  acquisition  program,  each  capability  is  increased  by 
the  gain  targeted  in  the  technology  policy.  The  process  then  repeats  again  with  the  next 
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acquisition  program.  This  yields  a  capabiiity  trajectory  for  the  technology  policy.  One 
example  is  shown  in  Figure  5. 


Deployed  Capability  T rajectory 
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Figure  5.  Sample  Capability  Trajectory 


We  see  in  Figure  5  that,  with  our  modei,  the  capabiiity  trajectory  is  a  step  function 
because  of  the  discrete  nature  of  acquisition  programs.  Thus,  we  see  that  the  longer  the 
acquisition  cycle,  the  longer  warfighters  must  make  due  with  older  equipment.  To  facilitate 
anaiysis,  we  would  like  to  capture  the  value  of  any  given  capability  trajectory  as  a  singie 
number.  We  wiil  do  so  through  the  average  deployed  capability.  To  caiculate  the  average 
deployed  capability,  we  select  a  time  horizon,  say  50  years,  and  then  calculate  the  average 
value  of  capability  over  that  time  intervai.  While  this  is  not  a  perfect  metric,  the  notion  we 
are  trying  to  capture  is  the  level  of  capability  that  warfighters  can  expect  from  their 
equipment  if  they  are  forced  to  engage  in  hostilities  without  warning.  This  allows  us  to 
compare  the  competing  strategies  of  small-but-rapid  capability  increments  versus  large-but- 
infrequent  capability  increments.  If  we  generate  many  sample  capability  trajectories  for  a 
particular  technology  policy,  we  can  calculate  the  expected  average  deployed  capability  to 
evaluate  the  efficacy  of  that  policy. 

Analysis  Results 

The  first  question  we  would  like  to  consider  is  whether  it  is  better  to  pursue  an 
evolutionary  vs.  a  revolutionary  technoiogy  strategy.  From  a  purely  performance  standpoint, 
we  can  answer  this  question  using  the  model  we  described  above  but  with  oniy  a  singie 
technology  for  each  acquisition  program.  To  make  this  more  concrete,  we  will  assume 
some  parameter  values  and  run  our  Monte  Carlo  simulation  over  a  range  of  technology 
policies.  In  particular,  the  range  we  consider  is  a  capability  gain  per  acquisition  cycie 
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between  10  and  30%.  The  relationship  to  the  upper  bound  of  the  duration  distribution  is 
described  in  Figure  4.  This  is  just  the  function: 

Function  1.  Capability  Gain  =  0.071841  (Upper 

Of  course,  other  functional  forms  are  possible,  and  we  will  discuss  these  in  more 
detail  later.  Under  this  function,  the  upper  bound  of  the  resulting  beta  distribution  can  vary 
between  2  and  20  years.  As  far  as  defining  the  rest  of  the  beta  distribution,  the  lower  bound 
is  always  2  years,  and  the  shape  parameters  are  a  =  2  and  p  =  5.  For  the  purposes  of 
calculating  the  average  deployed  capability,  the  initial  level  of  capability  is  always  one,  and 
the  time  horizon  is  50  years.  To  emphasize  the  impact  of  technology  development,  we  will 
assume  that  the  duration  of  the  system-integration  step  is  zero.  When  we  run  the  Monte 
Carlo  simulation  for  the  for  possible  technology  policies  within  the  range  of  10  to  30% 
capability  gain,  we  obtain  the  results  that  are  depicted  in  Figure  6. 


Performance  of  Techology  Policy 


Figure  6.  Performance  of  Technology  Policies  for  a  Single  Technology 


We  can  see  in  Figure  6  that  there  is  a  single  optimal  technology  policy  for  our 
performance  metric:  expected  average  deployed  capability.  In  fact,  the  optimal  policy  is  a 
relatively  modest  13.8%  target  improvement  in  capability  for  each  acquisition  cycle.  This 
policy  results  In  an  expected  average  deployed  capability  of  4.31.  This  result  seems  to 
suggest  that  from  a  performance  standpoint  it  is  better  to  take  smaller,  more  frequent  steps 
than  larger,  less  frequent  steps.  In  other  words,  evolutionary  is  better  than  revolutionary. 
But  is  this  always  the  case?  There  are  two  critical  features  of  this  model  that  we  can  vary. 
First,  we  can  alter  the  integration  time.  In  the  above  case,  it  was  set  to  zero.  But  if,  for 
example,  it  was  set  to  two  years,  the  single  technology  optimal  policy  increases  to  20%. 
This  result  is  reasonable  because  longer  integration  times  essentially  impose  more 
overhead  on  the  acquisition  process.  Consequently,  it  is  advantageous  to  target  a  larger 
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increase  in  capability  to  compensate  for  the  integration  delay.  However,  in  this  model, 
system  integration  is  not  linked  to  the  maturity  of  the  technology  selected.  In  some  cases,  a 
more  immature  technology  may  be  more  difficult  to  integrate  with  the  rest  of  the  system  and, 
hence,  would  actually  exacerbate  delays. 

Impact  of  Risk  vs.  Return 

The  second  feature  of  the  model  we  should  consider  is  the  shape  of  the  curve  that 
relates  risk  and  return.  The  function  used  in  our  model  is  displayed  in  Figure  4.  This  curve 
exhibits  the  diminishing  return  to  increasing  risk  that  was  mentioned  earlier.  But  what  would 
happen  if  the  penalty  for  additional  risk  were  more  severe?  In  other  words,  what  if  taking  on 
large  amounts  of  risk  resulted  in  very  little  gain  in  capability?  As  an  excursion,  we  will 
assume  that  the  relationship  between  the  gain  in  capability  and  the  upper  bound  of  duration 
is  determined  by  the  following  exponential  relationship. 

Function  2.  Capability  Gain  =  -0.811 04  6  °  supper  Bound  +0.300001 

We  find  that  under  this  risk-return  model,  the  optimal  single  technology  policy 
increases  to  26%.  If,  on  the  other  hand,  we  removed  the  diminishing  returns  to  risk  entirely, 
we  would  use  the  following  linear  relationship: 

Function  3.  Capability  Gain  =  0.01 1 1 1  l(Upper  Bound) +  0.077778 

Under  this  function  our  optimal  policy  is  10%,  the  minimum  allowable.  This  behavior 
is  perhaps  better  understood  visually.  Figure  7  shows  all  three  of  the  curves  discussed. 

Note  that  all  three  pass  through  the  same  maximum  and  minimum  points,  so  the  issue  is  just 
the  shape  of  the  curve.  Notice  also  that  the  exponential  curve  increases  sharply  then 
flattens  out.  The  high  initial  derivative  means  that  on  the  lower  end  of  the  curve,  one  can 
actually  gain  quite  a  bit  of  capability  for  very  little  risk.  But  the  curve  quickly  flattens  out 
such  that  each  additional  gain  in  capability  requires  a  huge  increase  in  risk.  Thus,  there  is  a 
natural  optimal  point.  The  same  is  true  for  the  baseline  curve.  While  it  is  not  as  severe, 
there  is  essentially  a  natural  optimal  increment  size.  For  the  linear  curve,  the  derivative  is 
constant,  so  the  best  strategy  is  to  minimize  the  size  of  the  increment.  In  this  extreme  case 
in  which  there  is  no  integration  time,  we  can  essentially  deploy  infinitesimally  small 
increments  of  capability  continuously.  Thus,  the  linear  case  ensures  that  the  best  possible 
capability  is  available  at  any  time.  While  this  case  would  be  desirable,  it  is  certainly  not 
realistic.  Something  akin  to  the  baseline  case  is  more  reasonable  because,  in  reality,  there 
is  usually  some  minimum  reasonable  increment  that  can  be  deployed. 
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Risk  -  Return  Tradeoff 


Figure  7.  Comparison  of  Different  Risk-return  Relationships 


It  is  important  to  note  that  changing  the  scale  of  the  risk-return  relationship  will 
certainly  change  the  optimal  policy,  but  here  we  focused  on  the  shape.  This  is  because  the 
shape  of  the  curve  is  what  determines  how  aggressive  the  optimal  policy  is  within  the 
feasible  ranges  of  capability  and  risk.  What  we  can  conclude  from  this  analysis  is  that,  from 
a  performance  standpoint,  there  is  a  natural  optimal  technology  policy,  and,  except  in 
extreme  circumstances,  that  policy  is  not  going  to  be  the  maximum  achievable  leap  in 
capability. 

Impact  of  Multiple  Technologies 

Thus,  for  a  single  technology  we  find  that  the  best  policy  will  most  likely  be  to  take 
small  steps  with  more  mature  technologies;  but  what  happens  when  a  program  depends  on 
the  integration  of  multiple  critical  technologies?  First,  we  will  assume  that  each  technology 
provides  a  different  capability.  For  example,  a  multi-mission  surface  combatant  would  have 
critical  technologies  that  provide  anti-air  and  anti-submarine  warfare  capabilities. 
Presumably,  stakeholders  for  each  area  or  capability  would  want  to  maximize  their 
respective  average  deployed  capability.  But  with  multiple  technologies  in  the  same 
program,  the  actions  of  one  affect  the  outcome  for  others.  For  example,  the  selection  of  an 
immature  technology  for  anti-air  warfare  could  delay  the  delivery  of  the  next  ship  class  and, 
consequently,  delay  the  deployment  of  the  next  generation  of  anti-submarine  warfare 
technology.  From  the  perspective  of  stakeholders  in  anti-submarine  warfare,  the  expected 
delay  means  that  if  they  must  wait,  they  should  target  a  larger  gain  in  capability  for  their  area 
to  compensate  for  the  delay.  But  since  program  completion  depends  on  both  technologies, 
the  reciprocating  decision  could  actually  exacerbate  delays  further.  In  order  to  understand 
stakeholder  behavior  when  a  program  incorporates  multiple  critical  technologies,  we  will 
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employ  game  theory.  (For  an  introductory  treatment  of  game  theory  see  Gibbons  (1992). 

For  a  more  advanced  treatment  see  Fudenberg  and  Tirole  (1991).) 

Game  theory  allows  us  to  consider  the  strategies  of  rational  competing  parties.  A 
technology  policy  would  be  the  targeted  percent  increase  for  each  capability  for  each 
acquisition  cycle.  For  example,  for  anti-air  we  might  target  a  15%  increase  per  cycle,  while 
for  anti-submarine  we  might  target  a  10%  increase.  Presumably,  stakeholders  for  each  area 
want  to  maximize  the  average  deployed  capability  for  their  area  of  concern.  To  employ 
game  theory,  we  must  find  the  best  response  functions  for  the  stakeholders  for  each  of  the 
capability  areas.  We  can  accomplish  this  by  finding  the  optimal  response  to  each  possible 
action  by  the  other  player.  Any  intersection  points  between  the  best  response  functions 
constitute  Nash  equilibria.  A  Nash  equilibrium  is  a  stable  point  in  strategy  at  which  either 
player  would  be  worse  off  if  they  deviated  from  that  strategy.  To  demonstrate  this  concept, 
we  will  assume  that  there  are  two  critical  technologies  in  each  acquisition  program.  Both 
have  the  identical  risk-return  behavior  from  the  baseline  case  above.  The  resulting  best 
response  functions  can  be  seen  in  Figure  8. 


Best  Response  Functions 


Figure  8.  Stakeholder  Best  Response  Functions  for  a  System  with  Two  Critical 

Technologies 

(The  intersection  of  the  two  functions  is  the  Nash  equilibrium.) 


The  plotted  points  in  Figure  8  represent  the  best  responses  over  the  selected 
policies.  Since  Monte  Carlo  simulation  was  used,  there  is  some  statistical  noise  in  these 
results.  Consequently,  a  linear  function  was  fit  to  the  best  response  data  for  each  player. 
We  can  see  from  the  best  response  functions  that  the  two  players  engage  in  reciprocating 
competition.  That  is,  as  each  player  Increases  his  targeted  capability,  the  best  response  of 
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the  other  player  is  to  increase  his  as  well.  Since  we  assumed  two  identical  players,  the 
Nash  equilibrium  is  symmetric  and  much  more  aggressive  than  what  is  optimal  for  a  system 
with  single  critical  technology.  In  fact,  the  equilibrium  solution  is  for  each  player  to  target  a 
23%  increase  in  capability  for  each  acquisition  cycle — resulting  in  an  average  deployed 
capability  of  2.7  for  each.  This  is  far  below  the  optimal  single  technology  result  of  4.31 .  The 
practical  implication  is  that  older  generations  of  technology  stay  in  the  field  much  longer. 

The  resulting  Nash  equilibrium  would  seem  to  corroborate  the  behavior  described 
previously.  If  one  player  chooses  a  particular  technology  policy,  it  is  in  the  best  interest  of 
the  other  player  to  choose  one  that  is  just  slightly  more  aggressive.  Consequently,  the  first 
player  might  as  well  choose  a  more  aggressive  one  himself,  and  so  on.  The  result  is  an 
equilibrium  state  with  a  much  more  aggressive  technology  policy  than  we  would  expect  from 
the  single  technology  analysis.  To  better  understand  this  result,  let  us  consider  the  case  in 
which  there  are  still  two  critical  technologies,  but  the  two  players  cooperate  in  selecting  a 
technology  policy. 

To  find  the  best  cooperative  technology  policies,  we  can  search  over  a  grid  of 
possible  policy  combinations.  The  results  are  plotted  in  Figure  9.  The  plotted  points  form 
the  space  of  all  possible  policy  outcomes.  Since  we  would  like  to  maximize  the  performance 
over  each  capability,  we  must  find  the  Pareto  optimal  set  of  polices.  A  Pareto  optimal  policy 
is  defined  such  that  to  improve  performance  of  one  capability  would  mean  sacrificing 
performance  on  another.  The  Pareto  optimal  set  is  designated  by  the  squares  in  Figure  9. 

Capability  Performance  Space 


Average  Capability  for  Technology  1 

Figure  9.  Performance  Space  of  all  Possible  Technology  Policies 

(The  squares  indicate  the  Pareto  optimal  set  of  policies.) 
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We  note  that  the  Pareto  optimal  frontier  allows  us  to  trade  off  some  performance 
between  the  two  capabilities;  but  even  so,  the  entire  frontier  is  superior  to  the  Nash 
equilibrium  achieved  in  the  non-cooperative  case.  For  the  sake  of  comparison,  let  us 
consider  the  optimal  symmetric  policy  from  the  frontier.  Under  this  policy,  the  target  is  a 
12%  improvement  per  acquisition  cycle  and  an  expected  average  deployed  capability  of 
3.99.  Note  that  this  is  a  significant  improvement  over  the  non-cooperative  solution  but  not 
as  good  as  the  single  technology  solution. 

What  does  this  result  tell  us?  We  can  conclude  several  things.  First,  the  best 
solution  for  a  program  that  relies  on  multiple  critical  technologies  is  a  cooperative  one  in 
which  a  small  amount  of  capability  is  sacrificed  from  each  area  to  bring  the  overall  cycle¬ 
time  down.  We  can  see  this  sacrifice  when  we  compare  the  optimal  symmetric  policy  to  the 
single  technology  policy.  Thus,  we  see  that  there  is  a  price  to  pay  for  including  multiple 
capabilities  in  a  single  system.  While  there  are  likely  cost  advantages,  there  will  be  some 
sacrifice  in  performance  (barring  synergistic  effects)  because  the  integration  of  multiple 
technologies  increases  acquisition  cycle-time.  More  importantly,  however,  is  that  the 
optimal  solution  Is  not  stable  in  that  it  is  not  a  Nash  equilibrium.  Therefore,  there  is  always 
an  incentive  to  deviate.  Let  us  say,  for  example,  that  we  select  the  optimal  symmetric 
technology  policy  for  our  system.  Assuming  that  everyone  else  follows  this  policy,  it  is  in  the 
best  interest  of  anyone  supporting  a  particular  capability  to  push  for  a  slightly  more 
aggressive  technology  for  his  area.  He  will  end  up  better  off.  But  since  all  have  an  incentive 
to  deviate,  if  one  deviates,  all  will  likely  deviate,  and  we  end  up  at  the  Nash  equilibrium. 

This  is  exactly  where  we  do  not  want  to  be. 

To  better  elaborate  on  this  point,  it  is  instructive  to  consider  the  cartel  problem  from 
economics.  In  a  cartel,  several  firms  make  a  price-fixing  agreement  so  that  they  can  all  earn 
greater  profits  than  if  they  competed.  Thus,  they  set  a  price  higher  than  the  market 
equilibrium  price.  However,  there  is  an  incentive  to  deviate.  If  one  firm  in  the  cartel  charges 
slightly  less  than  the  agreed-upon  price,  it  will  capture  the  market  and  make  much  more 
money  than  it  would  by  following  the  cartel  agreement.  Consequently,  cartels  tend  to  be 
unstable  without  strict  monitoring  and  enforcement. 

We  see  that  our  situation  here  is  quite  analogous.  For  a  given  system,  it  is  in  the 
best  interest  of  all  stakeholders  and  decision-makers  to  sacrifice  a  little  bit  of  capability  in 
each  critical  area  in  order  to  pursue  an  evolutionary  rather  than  a  revolutionary  policy. 
However,  it  is  always  in  the  best  interest  for  any  given  stakeholder  to  push  for  just  a  little  bit 
more  capability  in  his  respective  area.  Thus,  the  best  solution  is  unstable.  This 
phenomenon  could  explain,  at  least  in  part,  why  the  acquisition  system  in  the  Department  of 
Defense  consistently  pursues  revolutionary  rather  than  evolutionary  acquisition  programs 
despite  policies  to  the  contrary.  The  above  game  theory  analysis  indicates  that  in  the 
absence  of  enforcement,  the  rational  actions  of  decision-makers  with  good  intentions  will 
lead  to  poor  acquisition  policy.  The  implication  here  is  that  if  the  Department  of  Defense  is 
serious  about  evolutionary  acquisition,  it  cannot  expect  voluntary  compliance.  Compliance 
must  be  enforced. 

In  the  interests  of  robustness,  we  should  consider  the  sensitivity  of  this  result.  If  we 
increase  the  integration  time  to  two  years,  the  competitive  policy  is  a  29%  increase  in 
capability  per  cycle  for  an  average  deployed  capability  of  1 .78 — whereas  the  optimal 
symmetric  cooperative  policy  is  an  18%  increase  in  capability  per  cycle  with  an  average 
deployed  capability  of  2.05.  Thus,  an  increase  in  the  integration  delay  makes  both  policies 
more  aggressive,  but  the  relationship  between  competition  and  cooperation  is  preserved. 
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As  for  the  shape  of  the  risk-return  curve,  if  we  examine  the  exponentiai  case,  we  find  that 
the  competitive-cooperative  reiationship  is  stiii  preserved  but  becomes  iess  dramatic.  The 
competitive  policy  is  a  27%  increase  in  capabiiity  per  cycie  resulting  in  an  average  deployed 
capability  of  1 1 .7 — whereas  the  optimal  symmetric  cooperative  policy  is  a  24%  increase  in 
capability  per  cycle  with  an  average  deployed  capability  of  12.74.  (Note  that  these  average 
deployed  capabilities  are  very  high  because  the  exponential  curve  allows  for  large  increases 
in  capability  very  quickly.)  Finally,  what  happens  as  we  increase  the  number  of  critical 
technologies?  It  turns  out  that  the  situation  gets  worse.  If  there  are  three  critical 
technologies,  the  competitive  policy  is  to  pursue  a  33%  increase  in  capability  per  cycle  for 
an  average  deployed  capability  of  1.76.  Meanwhile,  the  optimal  symmetric  cooperative 
policy  for  three  identical  technologies  is  to  target  a  less-than-1 1  %  gain  in  capability  per  cycle 
resulting  in  an  average  deployed  capability  of  3.93. 

Thus,  the  key  result  from  this  analysis  is  that  when  an  acquisition  program  relies  on 
more  than  one  critical  technology,  the  relationship  between  competitive  and  cooperative 
behavior  is  fairly  robust.  The  cooperative  policy  yields  superior  performance  through  smaller 
capability  increments  but  is  unstable.  Without  enforcement,  the  situation  devolves  to  a 
suboptimal  Nash  equilibrium  that  achieves  inferior  performance  through  larger  capability 
increments. 

Cost  Considerations 

Up  until  this  point  we  have  only  considered  performance,  and  we  have  omitted  any 
discussion  of  cost.  An  evolutionary  approach  to  acquisition  may  achieve  a  higher  deployed 
performance  on  average,  but  is  it  more  cost-effective?  This  question  is  a  little  more  difficult 
to  answer;  it  depends  in  large  part  on  the  relative  costs  to  produce  and  deploy  the 
replacement  system  (or  upgrade  the  old  system)  at  the  end  of  each  cycle,  as  well  as  on  the 
relationship  between  technology  maturity  and  development  cost.  All  else  being  equal,  we 
can  say  that  there  is  a  tradeoff  between  cost  and  performance  in  terms  of  cycle-time.  More 
frequent,  shorter  cycles  mean  that  overhead  costs  associated  with  an  acquisition  cycle  are 
incurred  more  often.  Consequently,  costs  will  increase  when  the  cycle-time  is  shorter.  The 
relative  magnitudes  of  cycle  costs  versus  development  costs  will  dictate  the  severity  this 
tradeoff.  More  expensive  development  costs  reduce  the  contribution  of  cycle  costs  as  a 
percentage  of  the  overall  acquisition  bill.  Thus,  the  tradeoff  becomes  less  severe.  If,  on  the 
other  hand,  cycle  costs  are  very  high  (e.g.,  from  high  manufacturing  costs),  increasing  the 
length  of  acquisition  cycles  may  be  more  appealing. 

The  missing  piece  here  is  the  impact  of  technological  maturity  on  cost.  Does  the 
inclusion  of  immature  technology  in  an  acquisition  program  require  the  use  of  more 
expensive  development  methods  than  if  the  same  technology  were  pursued  in  a  research 
and  development  setting?  Does  the  inclusion  of  immature  technology  make  system 
integration  more  difficult  and  expensive  than  when  mature  technology  is  employed? 
Conventional  wisdom  would  suggest  that  the  answer  to  both  of  these  questions  is  yes,  and  if 
so,  there  could  be  an  optimal  technology  policy  that  minimizes  cost. 

We  can  demonstrate,  at  least  in  a  simplistic  way,  that  it  is  possible  to  achieve  lower 
costs  through  an  evolutionary  strategy.  Let  us  assume  that  we  have  a  system  with  a  single 
capability,  and  we  can  upgrade  that  capability  though  either  one  large  leap  or  multiple  small 
steps.  Both  achieve  the  same  end  capability,  but  increasing  the  number  of  cycles  to 
achieve  it  increases  the  maturity  of  the  technology  we  use  in  each  cycle.  For  example,  we 
could  achieve  a  25%  increase  in  capability  all  at  once  or  through  two  steps  that  sequentially 
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increase  capability  by  1 1 .8%  each  time.  The  capability  outcome  is  the  same  in  either  case, 
but  the  time  to  achieve  it  may  differ.  In  fact,  if  we  use  the  baseline  risk-return  relationship 
described  earlier,  the  one-step  strategy  is  expected  to  take  5.27  years  (assuming  no 
integration  time),  whereas  the  two-step  strategy  is  expected  to  take  4.46  years.  Of  course, 
the  addition  of  integration  time  could  erode  the  time  advantage  provided  by  multiple  steps, 
but  at  least  notionally  we  can  see  that  there  could  be  some  cost  advantage  to  taking 
multiple,  less-risky  steps.  If  we  assume  that  the  development  costs  are  the  same  in  both 
cases,  the  cost  difference  comes  down  to  the  overhead  associated  with  each  acquisition 
cycle.  Again,  for  the  sake  of  simplicity,  if  we  assume  that  the  cycle  costs  are  the  same 
regardless  of  the  aggressiveness  of  the  technology  policy,  we  can  determine  the  conditions 
under  which  the  evolutionary  strategy  is  more  cost-effective.  To  make  this  explicit,  let  us 
define  some  variables. 

n  =  the  number  of  steps  in  the  evolutionary  policy 

D(n)  =  the  expected  length  of  each  step  when  there  are  n  steps 

Cd  =  the  cost  rate  for  development  work 

Co  =  the  overhead  cost  for  each  acquisition  cycle 

As  above,  we  assume  that  we  have  two  policy  options  that  achieve  the  same 
increase  in  capability.  However,  the  first  policy  option  achieves  it  in  one  step,  while  the 
second  achieves  it  in  n  steps.  We  want  to  find  the  conditions  under  which  the  n-step  policy 
is  more  cost-effective  than  the  one-step  policy. 

Equation  1.  nD{n)C,^  +  hCq  <  D(1)Cd  -i-  Cq 
Rearranging  terms  yields: 


Equation  2.  — ^  < 


(D(l)-nD(n)) 

n-^ 


Thus,  we  can  characterize  the  cost-effectiveness  of  the  evolutionary  policy  in  terms 
of  the  ratio  of  the  cycle  costs  to  the  development  costs.  For  the  example  discussed  above, 
the  two-stage  policy  is  more  cost-effective  when  the  ratio  of  the  cycle  cost  to  the 
development  cost  rate  is  less  than  0.81 .  However,  this  analysis  is  admittedly  oversimplified 
and  only  reveals  the  possibility  that  an  evolutionary  strategy  could  be  less  expensive. 

Additional  work  is  needed  to  model  cost  in  a  more  realistic  manner.  Empirical 
studies  regarding  software  development  reveal  that  for  large  projects,  the  cost  savings  from 
a  spiral  approach  can  be  substantial  (Boehm,  2007).  This  occurs  because  the  upfront 
investment  in  better  systems  engineering  and  spiral  risk  reduction  is  outweighed  by  the 
reduction  in  rework.  This  is  especially  true  for  complex  software  systems.  This  result,  while 
particular  to  software  systems,  is  suggestive  and  may  imply  that  the  adverse  cost  impacts  of 
including  immature  technologies  in  a  single  acquisition  cycle  may  be  substantial.  Thus,  to 
effectively  model  cost,  we  must  consider  the  possible  approaches  to  developing 
technologies. 
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One  approach  is  to  leave  technologies  in  the  R&D  process  longer  so  that  they  are 
more  mature  when  they  are  finally  included  in  an  acquisition  program.  The  advantage  to 
this  approach  is  that  technologies  can  be  managed  in  a  portfolio  setting.  That  means 
funding  can  be  balanced  and  allocated  to  maximize  the  technological  options  available  to 
acquisition  programs.  If,  for  example,  in  the  course  of  development,  a  particular  technology 
proves  to  be  problematic  or  not  as  effective  as  anticipated,  funding  may  be  shifted  to  an 
alternate  approach  to  provide  a  needed  capability.  In  contrast,  once  technologies  are 
included  in  an  acquisition  program,  some  of  this  flexibility  is  lost.  There  is  a  great  deal  of 
commitment  to  a  particular  design  approach,  and  it  may  be  difficult  or  prohibitively 
expensive  to  change  it  in  the  event  that  a  selected  technology  underperforms.  Thus,  to 
really  model  the  cost  implications  of  evolutionary  acquisition,  one  would  need  to  model  the 
cost  impacts  of  withdrawing  technologies  from  the  R&D  portfolios  at  various  levels  of 
maturity.  This  must  be  relegated  to  future  work. 

Conclusions 

What  we  can  conclude  from  this  analysis  is  that,  from  a  performance  standpoint, 
every  acquisition  program  has  some  optimal  technology  policy  that  is  dependent  upon  the 
nature  of  the  system  and  technologies  involved.  Unfortunately,  the  implementation  of  this 
optimal  acquisition  strategy  is  not  trivial.  The  increased  emphasis  on  multi-mission  or  multi¬ 
capability  platforms  may  lead  to  overall  cost  savings  and  increased  flexibility,  but  it  creates  a 
tension  between  the  competing  missions  and  capabilities.  A  multi-mission  platform  means 
that  some  capability  must  be  sacrificed  relative  to  a  specialized  system  in  order  to  deliver 
the  system  in  a  reasonable  time  frame  and  to  maintain  the  optimal  acquisition  strategy.  The 
result  is  that  the  optimal  strategy  requires  an  unstable  technology  policy  that  incentivizes 
stakeholders  to  deviate  from  that  policy.  Thus,  there  is  a  tendency  in  the  Department  of 
Defense  to  pursue  an  overly  aggressive  technology  policy.  In  as  much  as  the  optimal  policy 
tends  to  be  more  moderate  than  the  stable  policy,  we  can  say  that  the  former  is  more 
evolutionary,  while  the  latter  is  more  revolutionary.  The  implication  is  that  while  evolutionary 
acquisition  is  more  appealing  from  a  performance  standpoint,  revolutionary  acquisition  is  the 
more  natural  outcome.  This  means  that  the  Department  of  Defense  cannot  expect  programs 
to  voluntarily  comply  with  evolutionary  acquisition  procedures  since  the  nature  of  the  system 
pressures  programs  towards  revolutionary  leaps  in  technology.  Consequently,  if  the  DoD  is 
serious  about  evolutionary  acquisition,  technology-maturity  requirements  must  be  strictly 
enforced. 
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Abstract 

The  defense  industry  has  witnessed  significant  consolidation  since  the  end  of  the 
Cold  War.  This  paper  explores  the  causes  of  the  wave  of  defense  mergers,  as  well  as  their 
impact.  The  analysis  finds  that  the  frequency  of  defense  mergers  is  more  strongly  correlated 
with  overall  merger  activity  in  the  economy  than  with  DoD  outlays.  In  examining  SAR  cost 
data  on  weapons  systems,  only  50-65%  of  the  weapons  systems’  costs  were  affected 
following  consolidation  activity  by  the  primary  contractor  that  made  them,  of  which  40%  of 
the  systems  experienced  a  statistically  significant  decrease  in  their  costs,  and  15-20% 
experienced  a  statistically  significant  increase.  Despite  a  2/3  reduction  in  the  number  of 
prime  contractors  in  the  fixed-wing  aircraft  sector  between  1990  and  1998,  about  60%  of  the 
systems  experienced  a  statistically  significantly  lower  cost  estimate.  For  the  tactical  missile 


^  The  author  gratefully  acknowledges  the  assistance  of  Maj.  Russell  Hoff  (US  Army)  and  Capt. 
Grisko  Alfonso  (USAF)  in  categorizing  the  weapons  systems  and  analyzing  the  SAR  data,  as  well  as 
the  hard  work  of  Lt.  David  Mazur  (USAF)  in  coliecting  the  SAR  data  and  Christina  Fishback  in 
providing  information  on  the  dates  of  the  defense  mergers.  The  author  appreciates  the  financial 
support  of  the  US  Naval  Postgraduate  School  Acquisitions  Research  Program.  The  views  in  this 
articie  represent  only  those  of  the  author  and  not  any  institution  with  which  the  author  is  affiliated. 
Please  do  not  quote  or  cite  without  permission  of  the  author.  See  contact  information  above. 
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category,  in  which  the  number  of  prime  contractors  also  fell  by  2/3,  28.6%  of  the  systems 
indicated  statistically  significantly  higher  post-merger  estimates  and  28.6%  of  them  indicated 
statistically  significantly  lower  post-merger  estimates.  Boeing,  Lockheed,  and  Raytheon 
were  among  the  few  main  primary  contractors  in  several  sectors  following  the  consolidation 
wave.  About  70-80%  of  the  weapons  systems  examined  in  this  analysis  which  were 
produced  by  them  indicated  a  statistically  significant  change  in  their  cost  estimates.  For 
Boeing  and  Lockheed,  50-57%  of  the  systems  exhibited  a  statistically  significant  reduction  in 
cost  estimates,  while,  for  Raytheon,  60%  of  the  systems  experienced  a  significant  cost 
increase.  About  2/3  of  the  systems  made  by  Lockheed  and  Martin  Marietta  manifested 
significant  cost  declines  following  the  Lockheed-Martin  Marietta  merger,,  and  about  %  of  the 
systems  made  by  Boeing  and  McDonnell  Douglas  experienced  a  statistically  significant 
decline  in  cost  estimates  following  their  merger.  This  suggests  that,  although  market 
concentration  levels  may  have  increased  in  certain  sectors,  DoD’s  costs  often  tended  to  be 
lower  in  the  post-merger  period  for  certain  weapons  systems. 

Introduction 

The  defense  industry  has  witnessed  significant  consolidation  since  the  end  of  the 
Cold  War.  As  the  number  of  large  defense  contractors  has  declined,  key  public  policy 
questions  have  arisen  concerning  whether  the  mergers  have  led  to  greater  efficiencies, 
lower  costs,  and  improvements  in  quality,  or  whether  they  have  led  to  higher  costs,  fewer 
choices,  and  larger  firms  with  unwieldy  organizational  structures.  The  purpose  of  this  paper 
is  to  examine:  (a)  the  roles  of  defense  spending  and  broader  merger  activity  in  the  economy 
on  the  frequency  and  size  of  defense  mergers,  (b)  the  patterns  of  defense  consolidation  and 
some  of  the  related  antitrust  concerns,  and  (c)  the  impact  of  mergers  of  major  defense 
contractors  on  the  costs  of  weapons  systems  facing  DoD. 

The  Impact  of  Defense  Spending  and  Broader  Merger  Activity 
on  Defense  Mergers 

The  wave  of  defense  mergers,  particularly  during  the  1990’s,  was  partially  driven  by 
the  need  to  eliminate  excess  capacity  in  the  industry  following  the  end  of  the  Cold  War. 
Overall  defense  spending,  as  well  as  defense  procurement  spending,  grew  rapidly  during 
the  1980’s,  declined  following  the  end  of  the  Cold  War,  increased  towards  the  end  of  the 
1990’s,  and  exhibited  significant  growth  with  the  War  on  Terrorism.  Indeed,  overall  defense 
spending  grew  73.5%,  and  defense  procurement  spending  grew  133.1%  between  1981  and 
1991,  while  between  1992  and  1996,  overall  defense  spending  fell  10.9%  and  defense 
procurement  spending  fell  34.7%.  Between  1997  and  2001,  overall  defense  spending  and 
defense  procurement  spending  grew  12.7%  and  15.3%,  respectively,  while  between  2002 
and  2006,  overall  defense  spending  and  defense  procurement  spending  grew  at  49.7%  and 
43.6%,  respectively.^  In  constant  FY  2001  dollars,  overall  defense  spending  declined  34.8% 
between  FY  1985  and  FY  1996  and  declined  25.6%  between  FY  1990  and  FY  1996. 


^  These  growth  rates  were  calculated  by  the  author  from  the  raw  data  in  the  Historical  Tables  (Table 
3.2)  for  the  United  States  Budget  for  Fiscal  Year  2008,  p.  56-60.  The  growth  rates  are  not  annualized 
nor  adjusted  for  inflation. 
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Defense  procurement  spending  declined  67.2%  between  FY  1985  and  FY  1996  and 
declined  53.77%  between  FY  1990  and  FY  1996.^ 

The  wave  of  mergers  in  the  defense  sector  was  also  partially  linked  to  overall  merger 
patterns  within  the  US  economy.  Table  1  shows  the  growth  rate  from  year  to  year  in  terms  of 
the  number  of  defense  mergers  and  the  value  of  defense  mergers,  as  compared  to  the 
comparable  growth  rates  for  merger  activity  in  the  US  economy. 

Table  1.  Annual  Growth  Rates  in  Merger  Activity  in  the  Defense  Sector  and  in  the 

Overali  Economy 


Time  Period 

Annual  growth 
rates  for  merger 
activity  (number  of 
transactions)  in 
the  defense  sector 

Annual  growth 
rates  for  merger 
activity  (number  of 
transactions)  in 
the  overall 
economy 

Annual  growth 
rates  for  merger 
activity  ($  value) 
in  the  defense 
sector 

Annual  growth 
rates  for  merger 
activity  ($  value 
)  in  the  overall 
economy 

1992-1993 

-44.83% 

4.008% 

-82.37% 

45.41% 

1993-1994 

-6.25% 

12.66% 

268.1% 

80.63% 

1994-1995 

-33.00% 

17.37% 

-94.13% 

30.94% 

1995-1996 

100.0% 

66.51% 

8571.4% 

110.8% 

1996-1997 

50.00% 

33.32% 

-46.96% 

35.68% 

1997-1998 

70.00% 

0.154% 

-59.25% 

83.41% 

1998-1999 

0.00% 

18.94% 

169.0% 

19.16% 

1999-2000 

-29.4% 

3.28% 

392.8% 

832.9% 

2000-2001 

-5.5% 

-13.37% 

-97.03% 

-94.72% 

2001-2002 

26.47% 

-12.06% 

164.7% 

-37.42% 

2002-2003 

-34.88% 

9.573% 

-55.97% 

15.14% 

2003-2004 

-10.7% 

22.66% 

50.50% 

48.78% 

I  NOTE:  These  annual  growth  rates  were  calculated  by  the  author  from  raw  data  found  in  the 
Mergerstat  Review  for  2005,  the  Mergerstat  Review  for  2002,  the  Mergerstat  Review  for  1997,  and 
the  Mergerstat  Review  for  1996.  The  defense  sector,  as  defined  by  Factset  Mergerstat,  encompassed 
firms  in  Standard  Industry  Classification  (SIC)  codes  3761-3769,  3721-3728,  and  3795.) 


Growth  in  merger  activity  in  the  defense  sector,  whether  measured  by  growth  in 
value  or  growth  in  number  of  transactions,  was  generally  lower  than  growth  in  merger 
activity  in  the  overall  economy.  Growth  in  merger  activity  in  the  defense  sector  exceeded 
growth  in  merger  activity  in  the  industry  overall  (or  exhibited  less  negative  growth)  in  terms 
of  the  number  of  transactions  and  in  terms  of  value  in  5  out  of  the  12  years  (41 .67%). 

Table  2  shows  the  number  of  defense  mergers  which  were  over  $100  million  in  value 
as  a  percentage  of  total  defense  mergers,  as  well  as  the  percentage  of  larger  mergers  of 
over  $100  million  in  size  in  the  economy  as  a  percentage  of  total  mergers  in  the  economy. 
The  years  in  which  large  defense  mergers  were  over  a  quarter  of  the  mergers  in  that  sector 


^  These  growth  rates  were  calculated  by  the  author  from  the  raw  data  in  the  Annuai  Report  to  the 
President  and  Congress  by  the  Secretary  of  Defense  in  2000,  Appendix  B-1 .  The  growth  rates  are 
calculated  from  data  in  constant  dollar  terms,  although  they  are  not  annualized. 
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were  1992,  1994,  1996,  and  2004.  In  the  overall  economy,  large  mergers  tended  to  be  a 
smaller  percentage  of  the  total  number  of  mergers  due  to  the  total  volume  of  mergers  during 
the  mid  to  late  1990’s.  Nevertheless,  the  fifth  column  suggests  that  the  period  of  the  most 
mega-mergers  in  the  economy  overall  stretched  from  1994-2000. 


Table  2.  Percentage  of  Defense  Mergers  and  Mergers  in  the  Overall  Economy 
Exceeding  $100  Million  in  Value 


Time  Period 

Number  of  $100m  plus 
transactions  as  a  percentage 
of  total  transactions  in  the 
defense  industry 

Number  of  $100m  plus 
transactions  as  a 
percentage  of  total 
transactions  in  the 
overall  economy 

1991 

0.00% 

8.01% 

1992 

27.59% 

7.54% 

1993 

18.75% 

9.03% 

1994 

40.0% 

12.64% 

1995 

0.00% 

13.2% 

1996 

40.0% 

10.84% 

1997 

20.0% 

11.16% 

1998 

19.6% 

1 1 .55% 

1999 

13.73% 

11.81% 

2000 

16.67% 

12.00% 

2001 

17.64% 

8.44% 

2002 

6.977% 

8.33% 

2003 

10.71% 

8.19% 

2004 

24.00% 

8.60% 

(NOTE:  These  percentages  were  calculated  by  the  author  from  raw  data  found  in  the  Mergerstat 
Review  for  2005,  the  Mergerstat  Review  for  2002,  the  Mergerstat  Review  for  1997,  and  the 
Mergerstat  Review  for  1996.  The  defense  sector,  as  defined  by  Factset  Mergerstat,  encompassed 
firms  in  Standard  Industry  Classification  (SIC)  codes  3761-3769,  3721-3728,  and  3795.) 

Industry  observers  often  cite  defense  spending  and  overall  merger  activity  as  the  two 
forces  behind  defense  sector  mergers  (Kerb,  1996).  But,  is  defense  merger  activity  more 
linked  to  the  level  of  DoD  spending  or  to  the  overall  level  of  merger  activity  in  the  economy? 
Which  one  of  these  is  a  more  significant  force?  Table  3,  which  shows  correlations  between 
various  measures  of  defense  merger  activity  and  merger  activity  in  the  overall  economy,  as 
well  as  between  defense  merger  activity  and  DoD  spending,  suggests  that  merger  activity  is 
much  more  strongly  linked  to  overall  activity  in  the  economy.  This  supports  the  hypothesis 
that  merger  activity  was  not  necessarily  entirely  driven  by  the  need  to  downsize  and  reduce 
excess  capacity  in  the  wake  of  the  Cold  War. 

The  correlations  use  data  covering  the  period  between  1992  and  2004.  Column  2 
shows  the  correlations  between  the  number  of  defense  mergers  in  a  given  year  and:  (a)  the 
overall  level  of  DoD  outlays  in  that  year,  (b)  the  level  of  DoD  procurement  outlays  in  that 
year;  (c)  the  overall  level  of  DoD  outlays  in  the  previous  year,  (d)  the  level  of  DoD 
procurement  outlays  in  the  previous  year,  and  (e)  the  level  of  overall  merger  activity  in  the 
economy.  Column  3  shows  the  comparable  correlations  for  defense  merger  activity  as 
measured  by  dollar  value,  rather  than  by  number  of  transactions. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  -  35  - 


Table  3.  Correlations  between  DoD  Outlays,  Merger  Activity  in  the  Economy, 
and  Merger  Activity  in  the  Defense  Sector 


Correlation  between: 

Number  of  defense  merger 
transactions  in  a  given  year 

Doilar  vaiue  of  defense  merger 
transactions  in  a  given  year 

Level  of  overall  DoD  outlays  in 
a  given  year 

-0.0269 

-0.2058 

Level  of  DoD  procurement 
outlays  in  a  given  year 

-0.3591 

-0.3783 

Level  of  overall  DoD  outlays  in 
the  previous  year 

-0.1929 

-0.2947 

Level  of  DoD  procurement 
outlays  in  the  previous  year 

-0.6097 

-0.3916 

Number  of  mergers  in  the 
overall  economy  in  a  given 
year 

0.6498 

Dollar  value  of  mergers  in  the 
overall  economy  in  a  given 
year 

0.9399 

(NOTE:  The  statistical  correlations  were  calculated  by  the  author  from  raw  data  found  in  the  Historical 
Tables  (Table  3.2)  for  the  Budget  for  Fiscal  Year  2008,  p.  56-50,  and  from  the  raw  data  found  in  the 
Mergerstat  Review  for  2005,  the  Mergerstat  Review  for  2002,  the  Mergerstat  Review  for  1997,  and 
the  Mergerstat  Review  for  1996.) 

The  correlations  between  defense  merger  activity  (regardless  of  how  it  is  measured) 
and  DoD  outlays  (regardless  of  whether  it  is  overall  levels  or  procurement  levels,  and 
whether  it  occurred  in  the  current  year  or  the  previous  year)  are  negative,  as  would  be 
expected — as  defense  spending  goes  down,  defense  merger  activity  goes  up.  Nevertheless, 
the  correlations  tend  to  be  weak.  Procurement  outlays  move  much  more  strongly  in  the 
opposite  direction  from  defense  transactions  than  overall  DoD  outlays  do.  Correlating 
previous  year  DoD  overall  outlays  and  procurement  outlays  with  current  year  merger  activity 
(in  terms  of  either  transactions  or  value)  yields  a  stronger  relationship  than  correlating 
current  year  outlays  with  current  year  merger  activity.  This  suggests  that,  since  the  merger 
process  requires  time,  mergers  are  a  delayed  response  to  spending  levels  in  previous 
years.  The  tightest  negative  relationship  is  between  merger  activity  (as  measured  by  the 
number  of  transactions)  and  DoD  procurement  outlays  in  the  previous  year. 

The  correlations  are  strongly  positive  between  merger  activity  in  the  defense  sector 
and  merger  activity  in  the  overall  economy  in  a  given  year  (excluding  defense  mergers) — as 
one  increases,  the  other  increases.  The  correlation  is  strongly  positive  between  the  number 
of  defense  mergers  and  the  number  of  mergers  in  the  economy  overall  (excluding  defense 
mergers)  at  0.6498,  while  the  correlation  is  very  strongly  positive  between  the  dollar  value  of 
mergers  in  the  overall  economy  (excluding  defense  mergers)  and  the  dollar  value  of  defense 
mergers  at  0.9399. 

In  summary.  Table  3  suggests  that  although  the  wave  of  defense  mergers  was 
driven  by  both  DoD  spending  and  by  overall  economic  merger  activity,  overall  economic 
merger  activity  was  much  more  strongly  correlated.  Consequently,  the  decline  in  Cold  War 
spending  and  its  impact  on  excess  capacity  was  less  important  than  overall  economic 
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growth,  stock  market  conditions,  and  the  need  for  defense  firms  to  defensively  merge  as 
their  rivals  merged  so  that  they  would  not  be  left  out  in  the  cold  as  a  relatively  smaller  firm 
facing  larger,  consolidated  competitors. 

Patterns  of  Defense  Consolidation  and  Antitrust  Concerns 

In  July,  1993,  Deputy  Defense  Secretary  William  Perry,  at  a  summit  known  as  the 
“Last  Supper,”  met  with  representatives  of  the  major  defense  contractors  and  encouraged 
significant  defense  sector  consolidation  (Ricks  &  Cole,  1998;  Cole,  1996).  Between  1990 
and  1998,  the  number  of  prime  contractors  decreased  significantly  due  to  consolidation  in 
10  of  the  12  key  defense  sectors  identified  by  DoD.  These  10  sectors  included:  tactical 
missiles,  fixed-wing  aircraft,  expendable  launch  vehicles,  satellites,  surface  ships,  tactical 
wheeled  vehicles,  tracked  combat  vehicles,  strategic  missiles,  torpedoes,  and  rotary-wing 
aircraft.  Table  4  shows,  for  each  of  the  10  sectors,  the  number  of  prime  contractors  in  1990, 
the  number  of  prime  contractors  in  1998,  and  the  amount  of  the  percentage  decline."^ 

Table  4.  Reduction  in  Prime  Contractors  in  Various  Weapons  Systems  Sectors 

between  1990  and  1998 


Sector 

Number  of  prime 
contractors  in  1990 

Number  of  prime 
contractors  in  1 998 

Percentage  reduction 

Tactical  Missiles 

13 

4 

-69.2% 

Fixed-wing  Aircraft 

8 

3 

-62.5% 

Expendable  Launch 
Vehicles 

6 

2 

-66.7% 

Satellites 

8 

5 

-37.5% 

Surface  Ships 

8 

5 

-37.5% 

Tactical  Wheeled 
Vehicles 

6 

4 

-33.3% 

Tracked  Combat 
Vehicles 

3 

2 

-33.0% 

Strategic  Missiles 

3 

2 

-33.0% 

Torpedoes 

3 

2 

-33.0% 

Rotary-wing  Aircraft 

4 

3 

-25.0% 

The  percentage  reduction  in  contractors  exceeded  60%  in  3  of  the  10  sectors,  and 
varied  between  25%  and  37.5%  in  the  remaining  7  of  the  10  sectors.  The  major  giants  which 
emerged  out  of  this  consolidation  across  these  sectors  were  Boeing,  Lockheed  Martin,  and 
Northrop  Grumman,  and,  to  a  lesser  degree,  Raytheon  and  General  Dynamics.  Between 
1990  and  1998,  the  three  sectors  which  experienced  the  most  consolidation,  and  which 
were  dominated  by  contractors  which  only  Included  Boeing,  Lockheed  Martin,  Northrop 


Data  on  the  sectors  and  the  number  of  contractors  in  1990  and  1998  are  derived  from  the  General 
Accounting  Office  (GAO)  Report  to  Congressional  Committees  on  the  Defense  Industry: 
Consolidation  and  Options  for  Preserving  Competition,  Washington  DC,  April  1998. 
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Grumman,  and  Raytheon,  were:  tactical  missiles,  fixed-wing  aircraft,  and  expendable  launch 
vehicles. 

By  1998,  Boeing  was  one  of  the  prime  contractors  in  6  of  the  10  markets:  tactical 
missiles,  fixed-wing  aircraft,  expendable  launch  vehicles,  satellites,  strategic  missiles,  and 
rotary-wing  aircraft.  Lockheed  Martin  was  one  of  the  prime  contractors  in  5  of  the  10  sectors: 
tacticai  missiles,  fixed-wing  aircraft,  expendabie  launch  vehicles,  satellites,  and  strategic 
missiles.  Northrop  Grumman  was  one  of  the  prime  contractors  in  3  of  the  10  sectors:  tactical 
missiles,  fixed-wing  aircraft,  and  torpedoes.  General  Dynamics  was  one  of  the  prime 
contractors  in  2  of  the  10  markets:  tracked  combat  vehicles  and  surface  ships.  Finally, 
Raytheon  was  one  of  the  prime  contractors  in  2  of  the  10  markets:  tactical  missiles  and 
torpedoes. 

With  the  increasing  numbers  of  defense  mergers  in  the  mid  to  late  1990’s,  the 
Antitrust  Division  of  the  Department  of  Justice  (DOJ)  and  the  Federal  Trade  Commission 
(FTC)  became  more  concerned  that  consolidation  was  leading  to  a  reduction  in  competition 
and  an  increase  in  anticompetitive  activity.  As  Joel  Klein,  Assistant  Attorney  General  of  the 
Antitrust  Division  of  the  DOJ  noted  in  his  address  before  the  Senate  Judiciary  Committee  in 
June,  1998,  “A  number  of  defense  mergers  proceeded  unchallenged  over  the  last  5  years, 
which  rationalized  capacity,  but  if  that  rationalization  goes  too  far,  it  can  harm  competition” 
(Klein,  1998).  Indeed,  the  DOJ  had  challenged  two  mergers  in  1997 —  Raytheon’s 
acquisition  of  Flughes  Aircraft  (the  aircraft  subsidiary  of  General  Motors)  and  Raytheon’s 
acquisition  of  the  defense  electronics  division  of  Texas  Instruments — but  then  allowed  both 
of  them  to  go  through  provided  that  divestitures  of  certain  key  divisions  occurred  prior  to  the 
merger  in  order  to  protect  competition.  In  1998,  however,  the  DOJ  blocked  the  merger 
between  Lockheed  Martin  and  Northrop  Grumman,  since  the  DOJ  believed  that  the  merger 
would  lead  to  a  reduction  in  competition  and  innovation  in  submarine  sonar  systems,  military 
aircraft  radar,  and  various  electronic  warfare  systems.  This  proposed  $1 1 .6  billion 
acquisition  was  the  largest  acquisition  that  the  DOJ  had  challenged  in  its  history  up  to  that 
point  (Klein,  1998),  and  the  challenge  was  supported  by  the  Pentagon  since  Defense 
Secretary  Cohen  also  thought  that  the  merger  would  be  anticompetitive  (Ricks  &  Cole, 

1 998).  Lockheed  and  Northrop  called  off  the  merger  in  July,  1 998,  prior  to  their  September 
trial  date  (Fidler  &  Lewis,  1998). 

Analyzing  the  anticompetitive  impact  of  consolidation  in  the  defense  sector  involves 
different  considerations  from  analyzing  consolidation  in  other  industries  for  several  reasons. 
First,  in  determining  the  relevant  geographic  market  of  possible  competitors,  the  analysis 
can’t  always  include  foreign  weapons  manufacturers  for  security  reasons,  although,  in  other 
industries,  foreign  manufacturers  can  be  included  in  defining  the  boundaries  of  the  market 
that  wouid  be  affected  by  the  merger.  Second,  traditional  industries  have  a  broader 
spectrum  of  consumers  for  the  product,  whereas  DoD  is  the  main  buyer  for  weapons 
systems.  Consequently,  it  plays  a  highly  significant  role  in  the  DOJ  and  FTC  deliberations. 
Third,  lower  barriers  to  entry  would  allow  new  entrants  to  enter  the  market  and  reduce  the 
possible  anticompetitive  effects  of  increased  consolidation,  such  as  higher  pricing. 
Nevertheless,  the  government  contracting  process  makes  it  harder  for  new  entrants  to  gain 
a  foothold  and  tends  to  give  an  advantage  to  incumbent  firms,  which  know  the  government 
contracting  system  better. 

Either  vertical  or  horizontal  consolidations  could  contribute  to  a  negative  outcome. 
Vertical  mergers  might  lead  to  foreclosure  to  competitors  of  key  input  suppliers  or 
distributors  along  the  vertical  supply  chain.  For  example,  one  of  the  concerns  about  the 
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proposed  Lockheed  Martin-Northrop  Grumman  merger  had  been  that  Lockheed  Martin 
would  have  control  of  a  key  supplier  of  electronics  which  supplied  Boeing’s  planes,  as  well 
as  its  own  planes.  This  could  enable  it  to  limit  Boeing’s  access  to  the  supplier.  On  the  other 
hand,  Lockheed  argued  that  the  Pentagon  could  monitor  the  selections  of  equipment  from 
outside  suppliers  and  that  the  process  was  sufficiently  transparent  that  this  would  not  be  an 
issue.  Indeed,  Lockheed  argued  that  the  mission  computers  in  its  F-16  planes  came  from 
Raytheon  (Ricks  &  Cole,  1998).  A  second  example  of  concerns  over  vertical  integration  was 
when  the  CEO  of  McDonnell  Douglas,  in  April,  1996,  announced  that  McDonnell  Douglas 
would  stop  buying  parts  from  Loral  for  its  jet  fighters  once  Lockheed  Martin  acquired  Loral. 
Paul  Kaminski,  the  chief  of  procurement  at  the  Pentagon,  wrote  to  McDonnell  Douglas, 
stating  that  this  could  ‘“increase  the  cost  or  lower  the  quality  of  the  products  you  supply’” 
and  that  if  the  best  product  is  offered  by  a  given  supplier,  which  ‘“happens  to  be  Loral,  then 
McDonnell  Douglas  should  continue  to  buy  from  that  company’”  (Cole,  1996). 

Horizontal  mergers,  in  the  absence  of  viable  international  competition  or  entry  by 
new  companies,  could  lead  to  increased  market  power  and  higher  prices  in  certain  sectors. 
For  example,  one  of  the  concerns  with  Raytheon’s  acquisition  of  Hughes  Aircraft  and  the 
defense  divisions  of  Texas  Instruments  in  1997  was  that  these  acquisitions  would  provide 
Raytheon  with  a  near  monopoly  position  in  spy  satellite  sensors,  night  vision  equipment,  and 
air-to-air  missiles.  Hughes  and  Raytheon  had  previously  been  strong  competitors  for  missile 
contracts,  and,  according  to  the  chief  of  acquisitions  at  the  Pentagon,  Paul  Kaminiski,  “their 
competition  saved  taxpayers  hundreds  of  millions  of  dollars,  shaving  70  percent  from 
Hughes’  original  price.”  Raytheon,  on  the  other  hand,  had  argued  that  other  companies  had 
competed  in  missile  competitions  and  had  won,  citing  McDonnell  Douglas’  and  Lockheed 
Martin’s  success  in  bidding  for  the  JASSM  missile  contract  (Mintz,1997). 

On  the  other  hand,  consolidation  might  also  lead  to  more  innovative  or  less  costly 
weapons  systems  due  to  greater  pooling  of  knowledge  between  consolidating  contractors. 
For  example,  Boeing,  which  had  acquired  Rockwell  and  McDonnell  Douglas,  succeeded 
over  Lockheed  in  winning  a  $5  billion  contract  for  a  National  Reconnaissance  Satellite  in 
1999.  At  the  time,  some  argued  that  the  combination  of  knowledge  and  talent  between 
McDonnell  Douglas,  Rockwell,  and  Boeing  enabled  the  unified  entity  to  win  the  contract  and 
that  this  would  not  have  been  possible  without  consolidation  (Flanigan,  1999).  A  second 
example  is  when  the  Navy  in  early  September,  1997  thought  that  the  proposed  merger 
between  Lockheed  Martin  and  Northrop  Grumman  would  have  actually  enabled  Lockheed, 
which  had  a  weaker  background  in  building  naval  aircraft,  to  compete  more  effectively 
against  Boeing  in  the  competition  for  the  new  Joint  Strike  Fighter  (Ricks  &  Cole,  1998).  The 
merger,  as  discussed  earlier,  did  not  take  place. 

Consolidation  activity  also  could  lead  to  improved  cost  efficiencies  from  reduced 
overhead  costs — combining  duplicative  facilities  and  corporate  headquarters,  rationalizing 
and  reducing  the  workforce,  pooling  R&D  funds,  and  more  effectively  using  pre-existing 
capacity.  Indeed,  when  the  Lockheed-Martin  Marietta  merger  took  place  in  1995,  it  was 
estimated  that  merging  telecommunications  operations,  research  divisions,  and 
headquarters,  would  save  $3  billion  over  the  following  five  years  (Mintz,  1994).  Some  of  the 
mergers  clearly  failed  to  yield  their  projected  saving,  however.  For  example,  Martin 
Marietta’s  1993  acquisition  of  General  Electric  Aerospace  had  only  yielded  half  of  the 
expected  cost  savings  three  years  later,  according  to  the  GAO  (Foote,  1996).  Two  years 
after  the  union  of  Hughes  Aircraft  and  General  Dynamics’  missile  division  in  1992,  the 
Inspector  General  could  not  verify  that  the  consolidation  had  saved  the  projected  $600 
million  for  the  Pentagon  (Korb,  1996). 
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Has  the  wave  of  defense  mergers  led  to  cost  savings  for  DoD?  According  to  the  Los 
Angeles  Times  in  October,  1999,  “Almost  a  decade  of  consolidation  in  the  defense  industry 
has  faiied  to  deliver  the  benefits  of  lower  costs  for  the  Pentagon.  And  the  mergers  of  the 
‘90’s  that  were  supposed  to  produce  stronger  and  more  innovative  defense  contractors  have 
more  often  caused  corporate  indigestion”  (Flanigan,  1999).  Industry  observers  argued  that 
innovation  had  suffered  from  the  mergers,  and  that  the  companies  had  become  too  big  and 
were  expending  significant  effort  in  managing  themselves  (Flanigan,  1999). 

The  issue  of  whether  DoD  recognized  cost  savings  from  the  wave  of  consolidation 
was  further  complicated  by  its  decision  to  pay  the  restructuring  costs  of  consolidation 
beginning  in  July,  1993,  provided  that  certain  conditions  from  the  consolidation  were  met, 
such  as  that  the  projected  savings  from  the  restructuring  would  exceed  the  costs.  Under  the 
1997  DoD  Appropriations  Act,  projected  savings  needed  to  exceed  costs  by  a  ratio  of  two  to 
one  for  business  combinations  occurring  after  September  30,  1996,  in  order  for  restructuring 
costs  to  be  reimbursed  (Cooper,  1997).  In  1997,  DoD  calculated  that,  through  September 
30,  1996,  for  every  $1 .00  that  it  paid  in  restructuring  costs,  it  estimated  $1 .93  in  savings 
because  it  had  paid  $179.2  million  in  restructuring  costs  and  realized  savings  of  $346.7 
miilion.  Nevertheless,  in  several  of  the  five  business  combinations  reviewed,  savings  was 
much  less  than  the  contractors  had  actually  estimated.  For  Lockheed  Martin,  the  estimated 
savings  used  to  certify  the  Lockheed-Martin  Marietta  merger  as  eligible  for  restructuring,  as 
of  September  30,  1996,  was  less  than  half  of  the  savings  estimate  which  had  originaily  been 
projected  (Cooper,  1997). 

Analysis  of  Cost  Data  on  Weapons  Systems  by  Type  and  by 
Defense  Contractor 

This  analysis  examines  whether  cost  estimates  for  weapons  systems  made  by 
leading  defense  contractors  increased  or  decreased  following  a  merger  with  another  major 
defense  contractor.  The  analysis  used  cost  data  from  the  summary  tables  in  the  Selected 
Acquisition  Reports  (SARS)  which  are  submitted  to  Congress  by  DoD  and  which  report  the 
acquisition  costs  of  Major  Defense  Acquisition  Programs  (MDAPS).^  Each  SAR  contains  a 
variety  of  various  items  on  the  mission  of  the  weapons  system  and  the  contractors  involved, 
as  well  as  data  on  the  costs  of  the  weapons  system,  including  baseline  cost  estimates  and 
quantity  estimates,  current  cost  estimates  and  quantity  estimates,  and  a  decomposition  of 
cost  changes  into  quantity  cost  changes,  schedule  cost  changes,  engineering  cost  changes, 
support  cost  changes,  estimating  cost  changes,  and  other  cost  changes.  The  period 
covered  in  the  SAR  data  used  in  this  analysis  encompassed  March,  1981  until  June,  2006. 

The  analysis  examined  28  weapons  systems/  programs;  this  is  only  a  subset  of  the 
weapons  programs  available  in  the  SARS.  These  systems  were  selected  because:  (a)  the 
primary  contractor  was  involved  in  a  merger  with  a  major  defense  contractor  during  the 


®  MDAP  (Major  Defense  Acquisition  Program) — “Defined  in  10  USC  §  2430  as  a  Department  of 
Defense  (DoD)  acquisition  program  that  is  not  a  highly  sensitive  classified  program  (as  determined  by 
the  Secretary  of  Defense)  and  that  is  designated  by  the  Secretary  of  Defense  as  a  major  defense 
acquisition  program,  or  that  is  estimated  by  the  Secretary  of  Defense  to  require  an  eventual  total 
expenditure  for  research,  development,  test,  and  evaluation  of  more  than  $365,000,000  (updated  to 
FY  2000  constant  dollars)  or  an  eventual  total  expenditure  for  procurement  of  more  than 
$2,190,000,000  (updated  to  FY  2000  constant  dollars)”  (Department  of  Defense,  2006,  August  3). 
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period  covered;  (b)  there  was  enough  time  series  data  to  examine  the  pre-merger  and  the 
post-merger  period;  (c)  the  weapons  system  was  only  made  for  one  of  the  services;  and  (d) 
the  contract  for  the  weapons  system,  during  the  period  covered,  did  not  have  a  defense 
contractor  that  was  not  involved  in  the  merger  as  its  primary  contractor.  The  research  is  still 
ongoing,  and  it  is  expected  that  more  weapons  systems/  programs  will  be  included  in  an 
expanded  version  of  this  preliminary  study. 

This  analysis  examines  the  current  year  cost  estimates  in  base  year  dollars  of  each 
weapons  system/program  over  time.  This  is  because  current  year  cost  estimates  in  base 
year  dollars  capture  overall  pre-  and  post-merger  effects  better  than  other  variables  in  the 
SARS,  which  decompose  the  cost  change  into  quantity  changes,  schedule  changes, 
engineering  changes,  etc.  A  merger  could  impact  cost  estimates  through  any  of  these 
avenues,  so  year-to-year  changes  in  overall  current  year  cost  estimates  in  base  year  dollars 
provided  the  best  measure.  An  expanded  version  of  this  preliminary  study  intends  to 
examine  the  other  components  of  the  cost  change  decomposition  in  greater  detail.  Current 
year  cost  estimates  in  base  year  dollars  were  also  used  to  minimize  the  impact  of  inflation. 

The  regression  model  used  for  each  of  the  28  weapons  systems/  programs 
regressed  current  year  cost  estimates  in  base  year  dollars  for  a  given  weapons  system  on  a 
time  trend  variable  and  on  an  indicator  variable  that  took  on  the  value  of  “1”  after  the  merger 
of  its  primary  contractor  and  “0”  before  the  merger.  The  time  trend  controlled  for  the 
increases  in  cost  estimates  over  time.  The  regression  model  appears  below: 

(Current  year  cost  estimates  in  base  year  doiiars),  =  a  +  Pi  (time  trend);  +  P2  (post¬ 
merger  indicator  variabie); 

The  regression  was  run  over  the  time  series  data  for  each  weapons  system.  In  one 
set  of  regressions,  the  post-merger  effect  was  assumed  to  take  place  beginning  with  the 
report  date  of  the  SAR  nearest  chronologically  to  the  effective  date  of  the  merger.  In  the 
second  set  of  regressions,  the  post-merger  effect  was  assumed  to  take  place  beginning  with 
the  report  date  of  the  SAR  which  was  the  second  nearest  chronologically  to  the  effective 
date  of  the  merger.  Although  the  timing  of  the  impact  of  a  merger  on  SAR  cost  estimates 
can  vary  between  contractors  and  weapons  systems,  the  analyses  focused  on  the  nearest 
SAR  to  the  merger  date  or  the  second  nearest  SAR  for  consistency. 

Tables  5  and  6  show  that  the  empirical  results  are  largely  robust,  regardless  of 
whether  the  post-merger  effect  is  assumed  to  occur  beginning  with  the  SAR  nearest 
chronologically  to  the  effective  merger  date  or  the  second  nearest  SAR  to  the  effective 
merger  date.  The  first  column  includes  the  name  of  the  weapons  system;  the  second 
column  gives  the  coefficient  (and  its  sign)  for  the  post-merger  indicator  variable;  the  third 
column  provides  the  p-value  for  the  statistical  significance  of  the  post-merger  effect  on  cost 
estimates;  the  fourth  column  gives  the  coefficient  (and  sign)  on  the  time  trend,  and  the  fifth 
column  provides  the  p-value  for  the  statistical  significance  of  the  time  trend. 

Table  5.  Regression  Results  with  the  Post-merger  Effect  Beginning  at  the  SAR 
Nearest  to  the  Effective  Date  of  the  Merger 


Weapons  System 

Coefficient  on  post¬ 

P-value  on 

Coefficient  on  time 

P-value  on 

merger  indicator 

coefficient  for 

trend  variable 

coefficient  for 

variable 

post-merger 

time  trend 

indicator 

variable 
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variable 

AH-64 

36.9611 

0.763 

47.257 

0.000 

AIM-9X 

1554.8 

0.000 

4.8778 

0.568 

ASAS 

-1419.66 

0.000 

16.395 

0.046 

AMRAAM 

-2826.00 

0.000 

183.26 

0.000 

ATACMS 

134.47 

0.366 

29.903 

0.000 

AV-8B 

-113.64 

0.001 

6.5453 

0.005 

ATCCS 

179.68 

0.046 

-12.833 

0.003 

ATICRM 

-49.355 

0.899 

64.324 

0.007 

C-17 

17687.66 

0.000 

319.77 

0.000 

DDG-51 

-6357.78 

0.001 

740.82 

0.000 

FA- 18 

-21133.99 

0.002 

635.6 

0.014 

F-22 

-8867.30 

0.151 

1074.1 

0.000 

Javelin 

-78.669 

0.840 

14.043 

0.291 

JDAM 

-669.47 

0.032 

147.651 

0.000 

JSOW 

542.25 

0.609 

-9.9954 

0.827 

JSTARS 

-1396.20 

0.003 

168.99 

0.000 

LHD-1 

251.02 

0.210 

53.764 

0.000 

Longbow  Apache 

-381.75 

0.612 

149.51 

0.000 

Longbow  Hellfire 

-759.73 

0.033 

36.382 

0.008 

NAVSTAR  User 
Equipment 

-212.399 

0.013 

29.502 

0.000 

Titan  IV 

-9604.985 

0.000 

504.366 

0.000 

DMSP 

15.714 

0.322 

6.557 

0.000 

FBCB2 

-422.658 

0.180 

4.646 

0.876 

MLRS 

-28.854 

0.744 

28.307 

0.000 

Strategic  Sealift 
Program 

58.530 

0.685 

20.624 

0.029 

T45TS 

143.59 

0.401 

47.809 

0.000 

Trident 

-2111.671 

0.056 

10.3506 

0.679 

JPATS 

744.526 

0.047 

124.02 

0.000 

Table  6.  Regression  Results  with  the  Post-merger  Effect  Beginning  at  the  Second 
Nearest  SAR  to  the  Effective  Date  of  the  Merger 


Lagged 

Coefficient  on 
post-merger 
indicator  variable 

P-value  on 
coefficient  for 
post-merger 
indicator  variable 

Coefficient  on 
time  trend 
variable 

P-value  on 
coefficient  for 
time  trend 
variable 

AH-64 

87.88 

0.48 

45.65 

0.000 
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AIM-9X 

1279.3 

0.000 

9.408 

0.422 

ASAS 

-1004.9 

0.002 

-8.205 

0.733 

AMRAAM 

-2953.6 

0.000 

184.6 

0.000 

ATACMS 

234.6 

0.108 

27.20 

0.000 

AV-8B 

-116.95 

0.001 

7.088 

0.004 

ATCCS 

194.91 

0.033 

-13.60 

0.002 

ATICRM 

255.64 

0.504 

49.295 

0.031 

C-17 

17138.7 

0.000 

336.68 

0.000 

DDG-51 

-7478.1 

0.000 

761 .47 

0.000 

FA- 18 

-24329.8 

0.000 

751.15 

0.003 

F-22 

-11220 

0.067 

1127.4 

0.000 

Javelin 

1156.99 

0.002 

-22.196 

0.067 

JDAM 

-698.65 

0.028 

149.39 

0.000 

JSOW 

1631.28 

0.126 

-50.687 

0.276 

JSTARS 

-1300.27 

0.005 

166.48 

0.000 

LHD-1 

144.32 

0.476 

55.225 

0.000 

Longbow  Apache 

-669.24 

0.372 

158.10 

0.000 

Longbow  Hellfire 

-789.56 

0.030 

38.132 

0.007 

NAVSTAR  User 
Equipment 

-191.89 

0.024 

28.756 

0.000 

Titan  IV 

-10094.5 

0.000 

513.14 

0.000 

DMSP 

30.865 

0.041 

5.910 

0.000 

FBCB2 

-606.34 

0.056 

22.475 

0.456 

MLRS 

-34.901 

0.693 

28.377 

0.000 

Strategic  Sealift 
Program 

93.856 

0.506 

19.345 

0.028 

T45TS 

63.6989 

0.707 

49.373 

0.000 

Trident 

-1489.63 

0.178 

-2.125 

0.933 

JPATS 

947.42 

0.006 

118.27 

0.000 

Table  7  summarizes  the  findings  of  Tables  5  and  6.  Again,  there  is  little  difference 
between  the  findings  if  the  merger  effect  is  assumed  to  begin  at  the  SAR  closest  to  the 
merger  effective  date  and  the  findings  if  the  merger  effect  is  assumed  to  begin  at  the  second 
nearest  SAR  to  the  merger  effective  date.  Between  54%  and  64%  of  the  systems  examined 
in  the  analysis  experienced  a  statistically  significant  change  in  their  cost  estimates  following 
a  merger,  controlling  for  the  time  trend.  Between  39%  and  43%  of  the  systems  experienced 
a  statistically  significant  negative  reduction  in  cost  estimates  in  the  post-merger  period, 
controlling  for  the  time  trend,  while  between  14%  and  21  %  of  the  systems  experienced  a 
positive,  statistically  significant  cost  increase.  This  suggests  that  defense  mergers  did  not 
always  experience  a  statistically  significant  change  in  their  cost  estimates  post-merger  but 
that,  for  those  systems  that  did,  the  cost  estimates  were  more  likely  to  decrease  than  to 
increase,  even  controlling  for  the  time  trend. 
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Table  7.  Percentage  of  Weapons  Systems  Experiencing  a  Post-merger  Change 

in  Cost  Estimates 


Percentage  of 
systems 
experiencing  a 
positive  and 
statistically 
significant  change 

Percentage  of 
systems 
experiencing  a 
negative  and 
statistically 
significant  change 

Percentage  of 
systems 
experiencing  a 
statistically 
significant  change 

Post  -merger 
effect  begins 
at  the  SAR 
closest  to  the 
merger 
effective  date 

14.3% 

39.3% 

53.6% 

Post-merger 
effect  begins 
at  the  second 
nearest  SAR 
to  the  merger 
effective  date 

21.4% 

42.9% 

64.3% 

Table  8  summarizes  the  weapons  systems  findings  from  Table  6  and  categorizes 
those  results  based  on  the  type  of  weapons  system  classification  found  in  the  1998  GAO 
report,  although  this  analysis  added  the  strategic  electronics  sector  and  the  munitions 
sector.  The  classification  of  the  weapons  systems  into  these  broader  categories  was  done 
by  examining  the  description  of  the  weapons  systems  in  the  SARS,  consulting  Jane’s, 
reading  materials  written  by  the  defense  contractors,  examining  The  2007-2008  Weapons 
Systems  from  the  Office  of  the  Assistant  Secretary  of  the  Army  for  Acquisitions,  Logistics, 
and  Technology,  and  reading  detail  on  each  system  written  by  the  Federation  of  American 
Scientists. 

The  categories  which  were  most  affected  by  the  mergers  in  the  sense  that  80-86%  of 
the  weapons  systems  in  those  categories  exhibited  a  statistically  significant  post-merger 
change  in  cost  estimates  were  the  strategic  electronics  category  and  the  fixed-wing  aircraft 
category.  About  57%-60%  of  those  systems  exhibited  a  statistically  significant  reduction  in 
cost  estimates,  controlling  for  the  time  trend.  Based  on  the  data  in  Table  4,  the  number  of 
prime  contractors  in  the  fixed-wing  aircraft  sector  experienced  a  62.5%  decline  between 
1990  and  1998.  Consequently,  this  analysis  suggests  that  although  market  concentration  in 
the  fixed-wing  aircraft  sector  increased,  this  led  to  more  significant  cost  decreases  than  cost 
increases  in  weapons  systems.  The  evidence  is  less  clear  in  the  tactical  missile  category,  in 
which,  based  on  the  data  in  Table  4,  the  number  of  contractors  declined  69.2%  between 
1990  and  1998.  About  57%  of  the  weapons  in  the  tactical  missile  category  exhibited 
statistically  significant  changes  in  their  cost  estimates,  of  which  28.6%  of  them  exhibited 
significant  increases  and  28.6%  of  them  exhibited  significant  decreases.  The  number  of 
prime  contractors  in  the  surface  ships  category  declined  37.5%,  but  the  only  system  in  that 
category  that  manifested  a  significant  change  exhibited  a  cost  decline.  The  analysis  had 
fewer  systems  in  the  rotary  aircraft,  strategic  missile,  munitions,  and  satellite  categories,  but 
a  subsequent  expanded  version  of  the  analysis  hopes  to  include  more  systems  in  these 
categories. 
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Table  8.  Percentage  of  Weapons  Systems  Experiencing  a  Post-merger  Change 
in  Cost  Estimates  by  Equipment  Type 


Percentage  of 
systems  in  each 
category  which 
experienced  a 
statistically 
significantly  higher 
cost  estimate  post¬ 
merger 

Percentage  of 
systems  in  each 
category  which 
experienced  a 
statistically 

significantly  lower  cost 
estimate  post-merger 

Percentage  of 
systems  in  each 
category  which 
experienced  a 
statistically 
significantly  different 
estimate  post-merger 
(higher  or  lower) 

Rotary  Aircraft 

AH-64 

Longbow  Apache 

0% 

0% 

0% 

Tactical  Missiie 

AIM-9X 

AMRAAM 

ATACMS 

Javelin 

JSOW 

Longbow  Hellfire 

MLRS 

28.6% 

28.6% 

57.1% 

Strategic  Electronics 

ASAS 

NAVSTAR  User 
Equipment 

FBCB2 

ATCCS 

ATICRM 

20% 

60% 

80% 

Fixed-wing  Aircraft 

AV-8B 

C-17 

FA- 18 

F-22 

JSTARS 

T45TS 

JPATS 

28.6% 

57.1% 

85.7% 

Surface  Ships 

DDG-51 

LHD-1 

Strategic  Sealift 
Program 

0% 

33% 

33% 

Satellite 

DMSP 

100% 

0% 

100% 
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Munition 

JDAM 

0% 

100% 

100% 

Strategic  Missiie 

0% 

50% 

50% 

Titan  IV 

Trident 

Table  9  summarizes  the  results  in  Table  6  by  defense  contractor.  Between  70%  and 
80%  of  the  weapons  systems  made  by  Boeing,  Raytheon,  and  Lockheed  experienced 
statistically  significant  changes  in  their  cost  estimates  following  their  mergers.  Raytheon  is 
the  only  one  of  the  major  contractors  which  had  a  higher  percentage  of  weapons  systems 
(60%)  that  experienced  a  statistically  significant  cost  increase  than  the  percentage  of 
weapons  systems  (20%)  that  experienced  a  statistically  significant  cost  decrease.  Over  half 
of  the  weapons  systems  made  by  Lockheed,  General  Dynamics,  and  Boeing  experienced  a 
statistically  significantly  lower  post-merger  cost  estimate.  As  discussed  earlier,  by  1998, 
Boeing  was  one  of  the  prime  contractors  in  6  of  the  10  markets,  and  Lockheed  Martin  was 
one  of  the  prime  contractors  in  5  of  the  10  markets.  Again,  this  evidence  suggests  that 
although  these  contractors  were  obtaining  greater  market  share  through  their  consolidation, 
the  mergers  were  more  likely  to  reduce  cost  estimates  for  the  weapons  systems  than  to 
increase  them.  Raytheon  is  the  exception,  but  it  was  one  of  the  prime  contractors  in  only  2 
of  the  10  markets  (as  delineated  by  the  1998  GAO  report)  and  so  had  less  opportunity  for 
market  power  than  Lockheed  Martin  and  Boeing. 


Table  9.  Summary  of  Statistically  Significant  Cost  Changes  by  Defense 

Contractor 


Percentage  of 
systems  made  by 
each  defense 
contractor  which 
experienced  a 
statistically 
significantly  higher 
cost  estimate  post¬ 
merger 

Percentage  of 
systems  made  by 
each  defense 
contractor  which 
experienced  a 
statistically 

significantly  lower  cost 
estimate  post-merger 

Percentage  of 
systems  made  by 
each  defense 
contractor  which 
experienced  a 
statistically 
significantly  different 
estimate  post-merger 
(higher  or  lower) 

Northrop 

0% 

40% 

40% 

Boeing 

14.3% 

57.1% 

71 .4% 

General  Dynamics 

0% 

50% 

50% 

Raytheon 

60% 

20% 

80% 

Lockheed 

25% 

50% 

75% 

McDonnell  Douglas 

14.3% 

42.8% 

57.1% 
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Table  10  explores  the  impact  of  the  merger  between  Lockheed  and  Martin  Marietta 
(effective  on  March  16,  1995)  and  the  merger  between  Boeing  and  McDonnell  Douglas 
(effective  on  August  1,  1997)  on  the  weapons  systems  produced  by  these  prime  contractors 
for  which  sufficient  data  was  available.  The  Lockheed-Martin  Marietta  merger  impacted  over 
80%  of  the  weapons  systems  examined,  but  2/3  of  them  experienced  a  statistically 
significant  decline  in  cost  estimates,  controlling  for  the  time  trend.  The  Boeing-McDonnell 
Douglas  merger  impacted  2/3  of  the  weapons  systems  examined,  of  which  50%  of  them 
experienced  a  statistically  significant  decline  in  cost  estimates,  controlling  for  the  time  trend. 

Table  10.  Impact  of  Selected  Defense  Mergers  on  Weapons  Systems  Cost  Estimates 


Percentage  of 
systems  made  by  the 
defense  contractors 
involved  in  a  specific 
merger  which 
experienced  a 
statistically 
significantly  higher 
cost  estimate  post¬ 
merger 

Percentage  of 
systems  made  by 
defense  contractors 
involved  in  a  specific 
merger  which 
experienced  a 
statistically 

significantly  lower  cost 
estimate  post-merger 

Percentage  of 
systems  made  by  the 
defense  contractors 
involved  in  a  specific 
merger  which 
experienced  a 
statistically 
significantly  different 
estimate  post-merger 
(higher  or  lower) 

Lockheed  /  Martin 
Marietta 

(March  16, 1995) 

ASAS 

F-22 

Longbow  Hellfire 

Titan  IV 

DMSP 

Trident 

16.7% 

66.7% 

83.3% 

Boeing/McDonnell 
Douglas 
(August  1,  1997) 

AV-8B 

C-17 

FA- 18 

JDAM 

Longbow  Apache 
T45TS 

16.7% 

50% 

66.7% 

Conclusions 

This  study  examines  evidence  on  the  causes  and  the  results  of  the  defense  merger 
wave  of  the  late  1990s.  Although  the  analysis  is  by  no  means  exhaustive,  it  does  suggest 
several  key  findings. 

First,  defense  mergers  are  negatively  correlated  with  DoD  procurement  outlays.  The 
correlation  between  defense  mergers  in  a  given  year  and  DoD  procurement  outlays  in  the 
previous  year  are  stronger  than  correlations  of  measures  in  the  current  year.  This  suggests 
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that  merger  activity  is  more  likely  to  be  a  delayed  response  to  previous  spending  ieveis  than 
to  current  spending  levels. 

Second,  the  correlations  between  defense  merger  activity  and  overall  merger  activity 
in  the  economy  are  strongly  positive.  On  balance,  the  correlations  between  defense  merger 
activity  and  overall  merger  activity  are  much  stronger  than  the  correlations  between  defense 
merger  activity  and  DoD  outiays.  This  suggests  that  merger  activity  was  driven  less  by 
declines  in  spending  foliowing  the  Cold  War,  and  more  by  a  stronger  economy  and  a  vibrant 
financial  market. 

Third,  the  reduction  in  the  number  of  prime  contractors  between  1990  and  1998  was 
more  substantiai  in  certain  sectors  than  in  others  and  resulted  in  some  of  the  defense 
contractors  becoming  dominant  across  sectors.  The  tactical  missiles,  fixed-wing  aircraft,  and 
expendable  launch  vehicle  sectors  experienced  a  2/3  reduction  in  the  number  of  prime 
contractors  during  the  period.  The  major  giants  which  emerged  from  the  consolidation  were 
Boeing  (one  of  the  prime  contractors  in  6  of  the  10  sectors),  Lockheed  Martin  (one  of  the 
prime  contractors  in  5  of  the  10  sectors),  and  Northrop  Grumman  (one  of  the  prime 
contractors  in  3  of  the  10  markets). 

Fourth,  in  examining  the  SAR  cost  data  on  28  weapons  systems,  only  50-65%  of 
them  exhibited  a  statistically  significant  post-merger  cost  change,  which  suggests  that  many 
weapons  systems’  estimates  were  unaffected  by  the  mergers.  About  40%  of  the  weapons 
systems  examined  in  this  anaiysis  experienced  a  statistically  significant  decrease  in  cost 
estimates,  controlling  for  the  time  trend,  and  about  15-20%  of  the  systems  experienced  a 
statistically  significant  increase  in  cost  estimates.  This  suggests  that,  to  the  extent  that  the 
weapons  systems  were  impacted  by  mergers,  a  greater  proportion  of  them  experienced  a 
reduction  in  costs  rather  than  an  increase  in  costs. 

Fifth,  when  the  weapons  systems  are  classified  into  the  10  categories  discussed  in 
the  1998  GAO  Report  (with  two  additional  categories),  the  fixed-wing  aircraft,  strategic 
eiectronics,  and  tacticai  missiie  categories  had  the  highest  percentage  of  systems  which 
experienced  a  statistically  significant  post-merger  change.  Within  the  strategic  electronics 
sector  and  the  fixed-wing  aircraft  sector,  about  60%  of  the  systems  experienced  a 
statisticaily  significantly  lower  cost  estimate  during  the  post-merger  period.  In  the  tactical 
missile  category,  28.6%  of  the  systems  surveyed  experienced  a  statistically  significantly 
higher  post-merger  cost  estimate  and  28.6%  of  the  systems  experienced  a  statistically 
significantly  lower  post-merger  cost  estimate.  This  suggests  that  in  the  fixed-wing  aircraft 
sector  especiaily,  which  manifested  a  2/3  decline  in  prime  contractors  between  1990  and 
1998,  the  increase  in  market  concentration  did  not  result  in  higher  costs  for  DoD.  The 
findings  were  evenly  split  in  the  tactical  missile  category,  which  also  experienced  a  2/3 
decline  in  contractors. 

Sixth,  when  the  weapons  systems  were  identified  with  their  primary  contractor, 
between  70%  and  80%  of  the  weapons  systems  examined  in  this  analysis  which  were 
produced  by  Boeing,  Raytheon,  and  Lockheed  experienced  a  statisticaiiy  significant  change 
in  their  cost  estimates.  For  Boeing  and  Lockheed,  50-57%  of  the  systems  experienced  a 
statistically  significant  reduction  in  cost  estimates.  Raytheon  was  the  only  contractor  for 
whom  60%  of  the  systems  experienced  a  statisticaiiy  significant  increase  in  their  cost 
estimates.  This  suggests  that  the  increases  in  market  power  may  not  have  translated  into 
higher  costs  for  DoD,  especially  for  systems  made  by  Lockheed  and  Boeing.  Indeed,  2/3  of 
the  systems  made  by  Lockheed  and  Martin  Marietta  experienced  a  statistically  significant 
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decline  in  cost  estimates  foiiowing  the  merger.  Half  of  the  systems  made  by  Boeing  and 
McDonneii  Douglas  experienced  a  statisticaily  significant  decline  in  cost  estimates  foliowing 
their  merger. 

In  conclusion,  the  analysis  suggests  that,  aithough  market  concentration  ieveis  in 
certain  sectors  increased  due  to  the  wave  of  defense  mergers,  DoD’s  costs  across  weapons 
systems  tended  to  be  lower  in  the  post-merger  period.  Although  further  research  on  a  larger 
sample  of  weapons  systems  distributed  across  various  sectors  is  necessary  to  more  fuily 
inform  the  pubiic  poiicy  discourse,  this  study  indicates  that  increases  in  market  power  do  not 
necessariiy  iead  to  an  anticompetitive  outcome  in  pricing.  Additionai  research  on  innovation 
cycles  within  the  weapons  systems  is  necessary,  as  weil  as  a  greater  assessment  of  the 
degree  to  which  international  competition  or  the  possibility  of  entry  of  smaller  competitors  in 
some  of  these  sub-sectors  constrained  cost  increases.  Many  of  the  questions  and  concerns 
in  the  eariier  rounds  of  consolidation  may  emerge  if  a  second  round  begins,  possibiy  at  a 
more  globai  ievei,  so  an  assessment  of  the  strengths  and  weaknesses  of  the  most  recent 
round  during  the  iate  1990’s  is  crucial. 
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Introduction 

Strategic  Sourcing  has  become  the  focus  of  buying  organizations  both  in  government 
and  the  private  sector.  In  the  federai  government,  the  Office  of  Federai  Procurement  Poiicy 
(OFPP)  began  an  initiative  in  2004  through  its  Chief  Acquisition  Officer  Councii  (CAOC)  to 
provide  for  Strategic  Sourcing  on  a  government-wide  basis.  Similarly,  State  and  Local 
governments  have  initiated  strategic  sourcing  initiatives  to  varying  degrees  across  the 
United  States.  In  support  of  this  initiative,  the  Nationai  Institute  of  Governmentai  Purchasing 
(NIGP)  adopted  a  resoiution  in  2006  supporting  the  use  of  Strategic  Sourcing  at  leveis  of 
government.  In  the  private  sector  Strategic  Sourcing  has  been  a  way  of  iife  now  for  over  a 
decade. 

For  the  uninitiated.  Strategic  Sourcing  has  become  synonymous  with  the  process  of 
identifying  certain  commodities,  or  commoditizabie  [sic]  services,  aggregating  those 
commodities,  and  then  driving  a  price  point  that  reflects  the  leveraged  buying  power  of  a 
particular  buying  organization.  Whiie  this  certainly  reflects  an  aspect  of  Strategic  Sourcing, 
Strategic  Sourcing  is  much  broader  and  encompasses  not  only  the  actual  sourcing 
decisions,  in  terms  of  aggregation  of  requirements  and  suppiier/supply-chain  management, 
but  also  the  analysis  of  ail  requirements,  the  capabiiities  of  the  suppiier  base  to  meet  those 
requirements,  and  the  appropriate  method  to  satisfy  requirements  through  the  suppiier  base, 
among  other  considerations.  In  some  cases,  managing  the  suppiier  base/supply  chain 
requires  the  identification  of  the  optimum  size  of  the  supplier  base/chain  and  the  mix  of 
companies  (read  capabiiities)  necessary  to  meet  the  demands  of  the  buyer.  On  occasion, 
the  management  of  the  supplier  base/chain  includes,  or  should  include,  efforts  to  expand  or 
contract  the  base,  or  the  mixture  within  the  base,  in  order  to  control  value  and  the  total  cost 
of  ownership  to  the  buyer.  The  ability  of  a  buyer  to  shape  the  base  is  dependent  on  the 
buyer’s  buying  power  and,  in  the  case  of  governmental  buyers,  other  policy  considerations 
associated  with  the  functions  of  a  sovereign.  It  is  from  this  iatter  perspective  that  we 
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approach  the  question  of  this  paper  of  whether  there  is  a  role  for  midsize  companies  in  the 
base/market  that  supports  government  acquisition  at  the  federal,  state  and  local  levels. 

Market  Shares  of  Smalls,  Midsize  and  Large  Companies 

At  the  moment,  the  absence  of  an  accepted  definition  for  midsize  companies  makes 
it  impossible,  with  any  precision  or  general  acceptance,  to  define  the  market  share  for 
midsize  companies.  In  fact,  the  current  statutory  and  regulatory  schema  divides  markets 
into  two  groups,  small  companies  and  other  than  small  companies.  Small  companies  are 
defined  by  statute  and  regulation  in  the  federal  market  and  in  most  state  and  local  markets. 
Even  where  small  companies  are  defined  by  statute  and  regulation,  there  is  room  for 
inaccuracy  precisely  because  of  the  way  in  which  the  definitions  are  drafted  and  as  result  of 
evolution  in  the  markets  themselves.  For  example,  the  size  status  of  a  company  in  the  US 
federal  market  is  determined  by  the  North  American  Industrial  Classification  System 
(NAICS),  which  establishes  size  status  based  on  the  code  selected.  The  various  codes  use 
one  of  two,  or  a  combination  thereof,  measurements  to  determine  the  size  of  a  company: 
sales  or  the  number  of  employees.  Because  the  federal  contracting  officer  determines  the 
NAICS  code  to  employ  in  a  particular  acquisition,  in  some  acquisitions  a  company  may  be 
small  and  in  others,  other  than  small.  In  addition,  in  the  federal  market,  there  are  process 
rules  that  allow  a  company  to  remain  small  for  reporting  requirements  for  the  duration  of  a 
contract,  even  though  after  submission  of  their  original  offer  an  event  occurred  that  caused 
the  company  to  become  other  than  small. 

It  has  often  been  argued  that  there  is  a  strong  strategic  and  economic  benefit  in 
encouraging  small  business  in  America.  The  Small  Business  Administration  (SBA) 
anticipates  that  small  businesses  will  be  responsible  for  the  nation’s  economic  growth  during 
the  coming  decades,  however,  at  great  risk  to  the  companies  themselves.  The  key  role  of 
small  enterprises  in  the  federal  market  space  has  further  been  codified  through  the  Small 
Business  Act  established  by  Congress.  The  purpose  of  the  Act  is  to  provide  assistance  to 
these  ventures  through  set-aside  programs,  sole-source  contracting  opportunities,  interest- 
free  loans  and  a  variety  of  government  funding  and  support  activities. 

In  the  information  technology  (IT)  arena,  the  welfare  of  small  and  midsized 
companies  is  critical  to  the  US  economy  in  that  the  birthplace  of  innovation  generally  is 
found  in  small  to  midsized  businesses,  and  the  new  technologies  will  open  avenues  for 
future  investment,  growth  and  employment  opportunities  for  the  future.  While  it  may  be  true 
that  large  companies  have  an  innovative  advantage  in  sectors  that  are  capital-intensive, 
highly  unionized  and  specialized,  small  businesses  have  the  innovation  advantage  when 
skilled  labor  comes  into  play  (Acs  &  Audretsch,  1987).  Secondly,  in  order  to  remain 
competitive  in  the  global  marketplace,  the  US  needs  to  constantly  feed  and  foster  new 
product  development. 

By  most  reports,  the  federal  government’s  SBA  is  serving  its  constituents  well. 
Marked  growth  in  some  industries,  especially  those  whose  market  sector  has  expanded  into 
global  commerce  via  the  Internet,  has  catapulted  the  previously  categorized  small  business 
into  the  midsize  sector.  The  IT  market  has  especially  fallen  prey  to  this  phenomenon  since 
the  late  1990s.  What  would  appear  to  be  a  boon  for  these  corporations  often  becomes 
detrimental  to  their  existence.  The  intended  protectionism  provided  by  the  Small  Business 
Act,  one  that  allowed  for  a  gradual,  steady  growth  of  a  business,  one  under  which 
experience  can  be  gained  and  expansion  revenue  generated,  is  then  lost  by  the  sudden 
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influx  of  economic  success  without  the  insight  and  wisdom  of  years  of  business- 
management  knowledge. 

Once  the  threshold  of  success  has  been  crossed  for  a  company  that  has  morphed 
from  small  to  midsize,  a  series  of  events  occur.  This  maturation  of  the  enterprise  ends  the 
exemption  provided  to  small  business  in  such  areas  as  cost  accounting  reporting  and 
governmental  contracting  compliance  standards.  These  emerging  enterprises  have  a 
greater  need  for  capital  for  expansion,  and  most  often  the  small  business  has  been  reliant 
on  the  credit  of  an  individual  such  as  the  owner  for  ready  cash.  As  the  small  business 
market  grows  into  midsize  success  stories,  the  risks  associated  with  achievement  are 
exponentially  multiplied  though  regulatory  requirements,  technology  exposures  and 
environmental  threats.  A  great  deal  of  companies  report  that  they  are  having  difficulties 
getting  financial  software  systems  that  will  meet  the  growing  needs  of  their  enterprise  at  a 
price  that  is  affordable  to  the  midsize  market.  This  exemplifies  the  challenge  of  finding 
something  stronger  than  Quicken  but  smaller  than  SAP  (large-corporation  ERP  system)  for 
their  finances.  Unfortunately  the  dummied-down  versions  are  not  available  in  the  mid¬ 
market  size.  This  begs  the  question  of  just  who  are  small,  midsize  and  large  businesses. 

Defining  Small,  Midsize  and  Large  Businesses 

In  the  market  generally,  small  businesses  are  viewed  by  the  public  as  those 
companies  that  have  less  than  100  employees  and  no  more  than  $500K  in  annual  sales 
(Peterson,  Albaum,  &  Kozmetsky,  1986).  This  general  interpretation  by  the  average  citizen 
differs  from  that  of  the  Small  Business  Administration’s  standards  of  a  maximum  of  500- 
1,500  employees  based  on  industry  type,  and  average  annual  sales  as  high  as  $17  million 
(Small  Business  Administration,  2006). 

In  the  management  accounting  software  realm,  the  middle  market  segment  is 
defined  as  companies  who  have  $1  million  to  $250  million  in  annual  sales — the  number  of 
employees  is  not  mentioned  as  the  regulations  and  accountability  of  such  revenue  dictates 
the  level  of  sophistication  needed  in  product  development  and  application. 

What  is  a  Midsize  Company? 

As  noted  earlier,  while  there  are  definitions  for  small  businesses  in  the  federal  market 
and  in  most  state  and  local  markets,  there  is  no  generally  accepted  schema  for  identifying 
midsize  companies.  There  clearly  is  no  agreement  on  how  a  company’s  size  should  be 
measured,  e.g.,  number  of  employees  or  sales  (gross  or  net).  Further,  there  is  an  even 
greater  divergence  of  opinion  at  what  point  a  company  other  than  small  becomes  a  large 
business  in  terms  of  number  of  employees  or  sales  (gross  or  net).  For  the  purposes  of  this 
discussion,  we  would  like  to  posit  that  the  definition  should  be  tied  to  the  gross  sales  of  the 
company  as  opposed  to  the  number  of  employees. 

We  suggest  the  use  of  gross  sales  as  a  measure  for  midsize  companies.  We  do  so 
in  no  small  part  as  a  reaction  to  how  the  markets  have  evolved  over  the  past  several 
decades.  At  one  point  in  time,  companies  integrated  their  functions  horizontally — 
performing  most  of  the  work  they  sold  within  their  own  employees.  Over  the  past  decades, 
companies  in  both  the  manufacturing  and  service  sectors  have,  to  a  great  extent, 
outsourced  non-core  functions  of  their  business,  keeping  as  members  of  their  in-house 
workforce  only  those  employees  key  to  the  core  functions,  and  relying  on  outside  sources  to 
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“ramp”  up  to  meet  business  engagements.  Using  the  number  of  employees  a  company  has 
in  this  environment  grossly  under  appreciates  the  share  of  the  market  a  company  may 
actually  control,  whereas  gross  sales  really  tell  the  story  of  a  company’s  dominance  in  a 
particular  market  sector. 

In  terms  of  what  threshold  should  be  used  to  determine  the  size  of  a  company  other 
than  small,  the  suggested  numbers  are  all  over  the  map.  The  US  General  Services 
Administration  (GSA),  in  preparing  a  Request  for  Proposal  (RFP)  for  its  next  generation 
Government-wide  Acquisition  Contract  (GWAC)  (named  Alliant),  issued  a  Request  for 
Information  (RFI)  from  the  IT  industry  (hardware,  software  and  services)  asking  for  feedback 
from  the  industry  on  what  the  cutoff  should  be  in  terms  of  sales  for  a  midsize  company  in 
that  market.  GSA  received  a  significant  amount  of  feedback  from  companies  with 
recommendations  ranging  from  $500  million  dollars  in  sales  and  up;  Yet,  the  majority  of 
those  responding  proposed  $500  million  in  sales  as  the  threshold. 

For  purposes  of  this  paper,  we  recommend  that  the  definition  be  set  at  $500  million 
in  gross  sales.  Over  time,  in  keeping  with  sound  Strategic  Sourcing  practices,  this  threshold 
should  be  reviewed  and  adjusted  as  the  nature  of  the  market  changes.  It  may  also  prove 
necessary,  as  the  SBA  found  for  small  businesses,  to  adjust  this  threshold  market  by  market 
based  on  specific  market  conditions.  Clearly,  this  would  prove  a  significant  effort  to 
undertake;  yet,  its  continual  maintenance  is  essential  to  achieve  sound  market  management 
to  meet  government  buyer  requirements. 

The  Importance  of  Midsize  Companies  to  a  Healthy 
Government  Market — Size  Really  Does  Matter 

What  happens  when  a  business  outgrows  its  competitors  and  is  forced  to  compete 
with  the  “big  dogs”  for  the  first  time — when  they  are  too  big  to  be  small  and  too  small  to  be 
big?  A  company  in  this  transitional  phase  of  moving  from  small  to  large  is  considered  to  be 
in  “no-man’s  land.”  It  is  here  that  they  have  maximized  the  ability  of  their  founders  to  further 
add  capacity  to  the  business,  while  at  the  same  time  burning  cash  in  order  to  keep  up  with 
the  market.  Additionally,  when  a  new  midsize  business  loses  a  major  client  it  had  been 
serving  during  its  evolution,  the  impact  of  such  a  loss  can  devastate  or  perhaps  destroy  the 
company  entirely  (Lafayette,  1992). 

Most  small  business  owners  sacrifice  themselves  for  the  betterment  of  the  business, 
putting  in  long  hours  and  taking  little  pay  for  their  efforts.  Often,  they  rely  on  the  skills  of  the 
founder,  and  when  capacity  dictates  additional  support,  they  find  that  they  must  pay  a 
prevailing  wage  for  the  work,  thus  taking  in  lower  profits. 

Along  with  the  additional  resources  needed  to  keep  up  with  demand  comes  the 
burden  of  growing  customer  expectations.  What  a  small  business  did  to  cater  to  its 
customers  (from  personalized  service,  friendly  conversations  and  lunch  meetings)  will  not 
only  be  expected  to  continue  as  the  firm  grows,  but  increasing  levels  of  service  will  be 
anticipated:  “Now  that  the  business  has  all  that  money  from  expansion,  well  surely  they 
could  have  house-calls,  and  free  replacements,  etc.” 

At  the  crux  of  this  explosion  of  opportunity  will  be  the  requirement  for  the  owner  to 
give  up  his  control  of  the  company.  A  CEO,  CFO  or  COO  will  be  needed,  and  this  is  not  a 
small  investment  to  be  had.  You  can’t  manage  a  larger  company  without  a  high-level 
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manager;  but  as  a  midsize  business,  you  may  not  be  able  to  afford  their  services.  Thus  the 
term  “no-man’s  land.” 


Cold  Hard  Cash 

As  the  personal  net  worth  of  the  founder  becomes  tied  to  the  company’s  balance 
sheet,  the  credit  of  the  company  becomes  risky.  There  appears  to  be  a  reluctance  to 
borrow  money,  as  repayment  causes  lower  profit  margins — already  very  tight  due  to  the 
competitiveness  of  the  marketplace  {Journal  of  Commerce,  1999,  January).  When  money 
is  needed  and  funds  are  hard  to  come  by,  this  is  when  the  decision  to  seek  venture  capital 
comes  into  play.  Control  of  the  corporation  by  these  investors  often  leads  to  the  separation 
of  the  founder  from  the  business  he/she  founded.  Further,  it  may  be  that  the  successful 
midsize  business  is  just  the  perfect  complement  to  the  large  corporation  seeking  to  acquire 
a  greater  market  share  and  lock-in  or  -out  its  competition.  In  the  end,  most  midsize 
companies  vanish  into  either  a  subdivision  of  a  larger  entity  or  disappear  completely  through 
dissolution. 


Better,  not  Bigger 

Some  companies,  when  faced  with  the  challenge  of  expansion,  have  discovered  that 
perhaps  the  growth  opportunity  is  not  worth  the  investment,  thus  choosing  to  be  better  and 
not  bigger.  These  companies  look  at  the  market  sector  in  which  they  operate  and  choose  to 
perfect  their  niche — moving  from  good  to  great  and  staying  in  control  of  their  own  destiny. 

In  situations  such  as  these,  the  most  important  motivators  for  the  company  are  found  not  in 
profit  generation,  but  in  relationships  that  have  been  built,  ties  that  have  been  created  in  a 
community  and  the  personal  rewards  that  have  generated  from  the  worthwhile  work  they 
perform  (Burlingham,  2006). 

Challenges  for  the  Midsize 

In  order  to  be  successful,  midsized  companies  must  be  able  to  foresee  the 
requirements  of  their  clients  and  find  innovative  methods  with  which  to  meet  these  needs.  A 
constant  stream  of  investment  must  always  be  made  in  which  cutting-edge  thoughts  and 
actions  are  encouraged  (Violino,  2006).  Companies  that  fail  to  take  action  in  these  areas 
tend  to  fail,  and  the  failure  to  keep  up  with  the  impact  of  IT  innovations  can  really  play  a 
critical  role  in  the  downfall  of  a  successful  company.  The  Internet  has  had  a  profound 
impact  on  companies  that  found  a  manner  in  which  to  incorporate  the  opportunity  of 
increased  exposure,  high  demand  and  customer  service.  Those  businesses  that  chose  to 
ignore  the  digital  revolution  are  now  awakening  to  a  business  model  that  fails  to  meet  the 
new  consumer  set  of  expectations. 

Risk  Tolerance 

The  amount  of  risk  tolerated  by  companies  is  often  dictated  by  their  size  and  market 
position.  Large  companies  tend  to  be  as  bureaucratic  and  risk  averse  as  governmental 
organizations.  Even  where  they  are  inclined  to  take  risk,  their  internal  processes  to  identify 
the  risk,  quantify  the  risk  and  get  approval  to  assume  it  are  lengthy  and  expensive.  Clearly, 
when  a  large  company  decides  to  assume  risk,  they  price  for  it  in  an  attempt  to  shift  as  large 
a  portion  as  possible  to  the  buyer.  Conversely,  small  companies  generally  lack  the 
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bureaucracy  associated  with  making  risk-assumption  in  large  businesses;  but  when  they 
face  the  “bet  the  company”  type  of  risk  issues,  their  inability  to  shift  the  risk  through  pricing 
or  to  bet  the  company’s  fate  on  a  single  acquisition  may  cause  them  to  pass  on  acquisition. 
Midsize  companies  are  particularly  situated  in  this  risk-tolerance  arena  to  take  on  the  risk 
without  having  to  bet  the  company  or  the  ability  to  fully  shift  the  risk  to  the  buyer  through 
pricing.  The  buyer  clearly  benefits  in  this  environment. 

Integration 

Another  arena  where  midsize  companies  provide  an  advantage  is  integration.  More 
and  more  today  government  buyers  find  themselves  in  situations  in  which  they  lack  the 
expertise  to  serve  as  integrator  for  solutions  they  require,  but  feel  uncomfortable  buying  the 
products  and  services  and  integration  function  from  a  single  provider.  Small  companies  will 
frequently  lack  expertise  in  integrating  these  complex  requirements,  although  they  are  more 
than  capable  of  providing  the  products  or  services  or  both.  Large  companies  are  capable  of 
providing  a  fully  integrated  solution,  but  experience  has  shown  that  it  may  be  to  the  buyer’s 
advantage  to  separate  the  integration  function  from  the  parts  to  be  integrated  to  create  a 
tension  in  the  supply  chain  that  ensures  that  decisions  are  made  that  ultimately  benefit  the 
buyer  while  focusing  on  performance  and  cost-benefit  trade-offs. 


Overhead  Costs 

Clearly,  an  advantage  that  midsize  companies  offer  (whether  in  fixed-priced  or  cost- 
type  acquisitions)  is  lower  overhead  rates.  Small  businesses  offer  similar  advantages  in  this 
area  as  well.  Lower  overhead  costs  benefit  the  buyer  in  both  fixed-price  and  cost-type 
contracts. 

Flexibility/Responsiveness 

The  advantages  in  flexibilities  and  responsiveness,  at  least  from  an  organizational 
perspective,  are  clear  to  any  observer  when  dealing  with  midsize  companies.  With  large 
companies,  try  calling  the  President  or  CEO  of  the  company  as  a  government  buyer  and 
getting  them  on  the  phone;  our  personal  experience  has  been  that  even  when  calling  to 
discuss  potential  suspension  or  debarment  (the  equivalent  of  the  death  penalty  in 
government  contracting),  getting  through  to  a  Large  company’s  senior  executives  to  talk  is 
difficult  at  best.  However,  at  the  midsize  level,  the  very  nature  of  the  organization 
allows/promotes  effective  communication  with  the  company’s  senior  executives. 

Similarly,  when  problems  arise,  even  the  best  contracts  with  the  best  companies  of 
all  sizes  have  problems;  midsize  companies  seem  to  demonstrate  greater  flexibility. 

Whether  this  flexibility  stems  from  the  lack  of  a  significant  internal  bureaucracy,  greater  risk 
tolerance,  relative  position  in  the  market  or  some  other  reasons,  is  irrelevant.  For  the 
government,  it  means  access  to  senior  management  and  more  timely,  effective  dispute 
resolution. 

Boom  and  Bust  Cycle 

Finally,  there  is  a  boom  and  bust  cycle  in  government  marketplaces.  Government  at 
all  levels  promotes  the  development  of  small  businesses  and  supports  their  growth  through 
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programs  designed  to  help  them  grow  in  size  and  gain  experience  in  performing  work. 
However,  once  they  graduate  to  “other  than  small”  in  the  US  market,  the  government 
support  network  disappears  completely,  leaving  most  of  these  companies  in  the  situation 
where  they  lack  the  government  support  programs  to  sustain  their  continued  growth  and,  as 
noted  above,  the  capital  to  compete  with  large  companies.  Thus,  many  either  find 
themselves  selling  their  company  or  going  out  of  business.  This  is  an  incredibly  wasteful 
process  in  which  the  company  that  the  government  invested  in  when  it  was  small  because  it 
was  the  strongest  part  of  the  engine  of  our  economy  both  in  terms  of  producing  jobs  and  in 
terms  of  technology  development,  patent  submissions,  etc.,  simply  goes  away — and  along 
with  it,  often  the  jobs  it  created,  the  innovation  it  promoted  and  the  experience  acquired 
during  the  time  it  enjoyed  the  protections  of  the  various  small  business  programs.  By 
identifying  a  mid-tier  in  the  market  and  managing  that  tier  so  companies  do  not  have  to  sell 
or  go  out  of  business,  waste  is  eliminated  and  an  overall  stronger  economy  results — one 
better  capable  of  surviving  market  shifts. 

Conclusion 

There  is  unquestionably  a  dearth  of  empirical  data  to  support  the  suggestions  in  this 
paper.  The  dearth  exists  not  because  the  data  isn’t  available  but  because  government 
markets  have  not  focused  on  it  to  date.  Intuition  informs  many  that  there  is  an  important  role 
for  midsize  companies  in  meeting  the  requirements  of  the  government  market,  particularly  in 
the  challenging  times  facing  government  buyers  today  both  domestically  and  internationally. 

Midsize  companies  offer  lower  cost  solutions,  greater  risk  tolerance  and  more 
responsiveness  than  large  companies;  combine  that  with  greater  experience  than  small 
companies  have  had  the  opportunity  to  garner,  and  it  is  clear  that  there  is  a  value  to  the 
government  buyer  in  having  midsize  companies  in  the  government  market  place... 
Government  buyers,  however,  lack  the  empirical  data  they  need  to  utilize  existing 
procurement  flexibilities  to  manage  government  markets  to  ensure  that  a  sufficient  number 
of  midsize  companies  remain  available  to  meet  government  requirements. 

What  is  needed  at  this  point  is  greater  study  of  the  markets,  empirical  demonstration 
of  the  value  of  midsize  companies  to  those  markets  and  then  the  development  of  an 
effective  program  designed  to  ensure  that  a  sufficient  mix  of  companies  exist  to  meet  the 
government’s  requirements  in  a  timely  manner  and  at  best  value. 
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Abstract 

This  study  investigated  the  relationships  between  self-leadership  and  creativity  in  the 
context  of  a  defense  acquisition  organization.  More  specifically,  this  study  examined 
differences  in  self-leadership,  creativity  and  perceived  organizational  support  for  creativity 
between  line  and  supervisory  defense  acquisition  employees.  Our  analyses  suggested  that 
self-leadership  was  significantly  related  to  creative  potential  and  practiced  creativity  for  both 
line  and  supervisory  employees,  although  there  were  no  significant  differences  in  overall 
levels  of  self-leadership  between  the  two  groups.  In  addition,  we  found  significant 
differences  in  creative  potential,  practiced  creativity,  gap  scores  and  perceptions  of 
organizational  support  for  creativity.  Specifically,  line  employees  reported  significantly  lower 
levels  of  creative  potential,  practiced  creativity  and  perceptions  of  organizational  support  for 
creativity  along  with  higher  gap  scores  in  comparison  to  supervisors. 

Our  findings  imply  that  self-leadership  is  a  primary  tool  for  facilitating  creativity  at  all 
organizational  levels  and  that  active  organizational  support  for  creativity  may  be  the  key  for 
reducing  the  gap  between  creative  potential  and  practiced  creativity  that  represents 
untapped  creative  resources.  Our  results  suggest  that  this  gap  is  much  more  pronounced 
among  line  employees  and  that  line  employees  generally  perceive  less  organizational 
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support  for  utilizing  their  creative  resources  than  supervisors.  In  response,  we  make  some 
specific  suggestions  for  organizational  interventions  designed  to  increase  self-leadership 
capabilities  at  all  levels  and  to  increase  perceptions  of  organization  support  for  creative 
practices  among  line  employees  in  defense  acquisitions.  A  workforce  with  creative  self- 
leaders  could  synergistically  assist  organizations  in  maximizing  the  utility  of  all 
organizational  resources. 

Introduction 

Innovation  and  creativity  are  critical  for  organizations  to  thrive  in  the  21  century 
(Kanter,  1983;  Tushman  &  O’Reilly,  1997;  Utterback,  1994).  Indeed,  the  Office  of  Force 
Transformation  (OFT)  under  the  Office  of  Secretary  of  Defense  (OSD)  has  placed  the 
leveraging  of  innovation  and  creativity  among  the  most  effective  approaches  for  creating  the 
transformational  changes  needed  to  maintain  the  Department  of  Defense’s  strategic 
position.  Creativity  is  more  likely  to  occur  If  an  individual  has  certain  characteristics  or  innate 
skills  and  abilities  (Tierney  &  Farmer,  2002;  Hinton,  1970;  Simonton,  1992;  Woodman  & 
Schoenfeldt,  1989)  and  when  the  individual  perceives  that  the  work  environment  supports 
creativity  (Amabile,  1996;  Cummings,  Hinton  &  Gobdel,  1975;  Woodman,  Sawyer  &  Griffen, 
1993).  Furthermore,  the  ability  to  leverage  creativity  depends  largely  on  effective  leadership 
(Kouzes  &  Posner,  1995;  Manz  &  Sims,  2001).  A  common  theme  in  improving  leadership 
effectiveness  concerns  knowing  and  leading  oneself  (Bennis,  1994;  Drucker,  1999; 

Goleman,  Boyatzis  &  McKee,  2002;  Senge,  1990;  YukI,  2002).  Self-leadership  is  a  concept 
that  focuses  on  self-reflection  and  evaluation  aimed  at  improving  personal  and  professional 
performance. 

Although  theorists  have  often  suggested  relationships  between  self-leadership  and 
creativity  (e.g.,  Kouzes  &  Posner,  1995;  Manz  &  Sims,  2001),  very  little  attention  has  been 
given  to  how  these  relationships  may  differ  across  organizational  levels.  The  purpose  of  the 
current  study  is  to  examine  the  self-leadership  and  creativity  differences  in  line  and 
supervisory  defense  acquisition  employees.  The  differences  identified  in  this  research  may 
have  important  implications  for  maximizing  employee  self-direction  and  for  fully  utilizing 
creative  resources  at  all  organizational  levels. 

Creativity  and  Seif-leadership 

Although  creativity  is  a  complex  concept  that  Is  somewhat  difficult  to  define, 
consistent  themes  tend  to  emerge  across  the  various  definitions  in  the  creativity  literature 
(e.g.,  Barron  &  Harrington,  1981;  Guilford,  1950;  Martindale,  1989;  Sternberg  &  Lubart, 
1999).  Based  on  the  common  ideas  in  these  definitions,  we  define  creativity  as  an  ability  to 
harvest  novel  but  appropriate  ideas  in  order  to  maximize  efficiencies,  solve  problems,  and 
increase  effectiveness.  We  further  divide  the  creativity  concept  into  creative  potential  and 
practiced  creativity  (e.g.,  Hinton,  1968;  DiLiello  &  Houghton,  2006,  2007).  In  short,  if  an 
individual’s  creativity  is  attenuated  by  the  environment,  then  the  individual  will  not  utilize  his 
or  her  full  creative  potential  (Hinton,  1968;  George  &  Zhou,  2001;  Scott,  1965). 

Creative  potential  is  the  creative  capacity,  skills  and  abilities  that  a  person  possesses 
(Hinton,  1968,  1970).  Creative  potential  includes  the  concept  of  creative  self-efficacy,  an 
individual’s  subjective  assessment  of  their  personal  ability  to  be  creative  (Tierney  &  Farmer, 
2002).  Creative  self-efficacy  involves  seeing  oneself  as  being  good  at  creative  problem¬ 
solving  and  generating  novel  ideas.  Creative  potential  also  includes  having  the  talent  or 
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expertise  to  do  well  in  one’s  work  and  possessing  the  ability  to  take  risks  by  trying  out  new 
ideas  (Amabile,  Burnside&  Gryskiewicz,  1999). 

Practiced  creativity,  on  the  other  hand,  is  the  perceived  opportunity  to  utilize 
creativity  skills  and  abilities.  Practiced  creativity  should  not  be  confused  with  creative 
performance,  which  is  an  external  assessment  of  products  or  achievements  (Amabile,  1996; 
Hinton,  1968).  Practiced  creativity  is  also  different  from  the  concept  of  organizational 
support  for  creativity,  which  is  the  extent  to  which,  “an  organizational  culture  [...]  encourages 
creativity  through  the  fair,  constructive  judgment  of  ideas,  reward  and  recognition  for 
creative  work,  mechanisms  for  developing  new  ideas,  and  active  flow  of  ideas,  and  a  shared 
vision  of  what  the  organization  is  trying  to  do”  (Amabile  et  al.,  1999,  p.  15).  Employees  with 
strong  creative  potential  are  more  likely  to  actually  practice  creativity  when  they  perceive 
strong  support  from  the  organization  (DiLiello  &  Houghton,  2006);  several  key  conditions 
must  be  present  within  an  organization  for  its  work  environment  to  support  individual 
creativity  (e.g.,  Amabile,  1988;  Ford,  1996;  Mumford  &  Gustafson,  1988). 

The  distinction  between  creative  potential  and  practiced  creativity  is  important 
because  when  people  perceive  themselves  as  having  creative  potential  but  do  not  perceive 
the  ability  to  use  or  practice  this  potential,  they  will  be  less  likely  to  engage  in  creative 
behavior.  The  gap  between  creative  potential  and  creative  practice  represents  untapped 
organizational  resources.  Identifying  such  untapped  resources  may  be  especially  important 
in  defense  acquisition  organizations  that  are  continually  being  told  to  “do  more  with  less.” 

Self-leadership  (e.g.,  Manz,  1986;  Neck  &  Houghton,  2006;  Neck  &  Manz,  2007)  is  a 
self-evaluation  and  self-influence  process  through  which  individuals  identify  and  replace 
ineffective  behaviors  and  negative  thought  processes  with  more  effective  behaviors  and 
positive  thought  processes,  thereby  enhancing  personal  accountability  and  improving 
professional  performance.  Theorists  have  long  suggested  that  leaders  in  organizations 
should  encourage  their  followers  to  lead  themselves  in  the  workplace  (e.g.,  Manz  &  Sims, 
1980,  2001).  Supervisors  and  work  environments  only  have  a  limited  control  over  the 
workers;  additional  control  or  work  motivation  must  come  from  within  the  individual 
(Herzberg,  Mausner  &  Snyderman,  2003;  Manz  &  Sims,  1980;  Sergiovanni,  1992).  When 
employees  are  trained  and  empowered  to  lead  themselves,  supervisors  can  shift  their  focus 
from  detailed  oversight  and  control  to  longer-term,  big-picture  issues. 

Founded  upon  several  classic  theories  of  self-influence — including  self-regulation 
(Kanfer,  1970;  Carver  &  Scheier,  1981),  self-control  (Cautela,  1969;  Mahoney  &  Arnkoff, 
1978,  1979;  Thoresen  &  Mahoney,  1974),  intrinsic  motivation  theory  (e.g.,  Deci  &  Ryan, 
1985),  and  social  cognitive  theory  (e.g.,  Bandura,  1986) — self-leadership  is  a  normative 
model  that  prescribes  specific  sets  of  behavioral  and  cognitive  strategies  aimed  at 
increasing  individual  performance.  Self-leadership  strategies  are  often  divided  into  three 
primary  categories:  Behavior  Focused  Strategies,  Natural  Reward  Strategies  and 
Constructive  Thought  Strategies  (e.g..  Neck  &  Houghton,  2006). 

Behavior  Focused  Strategies  involve  identifying  and  replacing  ineffective  behaviors 
with  more  effective  ones  through  a  process  of  self-observation,  self-goal  setting,  self-reward 
and  self-correcting  feedback  (Neck  &  Houghton,  2006).  Self-observation  entails  a  close 
examination  of  one’s  own  behaviors  in  order  to  identify  behaviors  that  should  be  changed, 
enhanced  or  eliminated  (Mahoney  &  Arnkoff,  1978,  1979;  Manz  &  Sims,  1980;  Neck  & 

Manz,  2007).  Once  target  behaviors  have  been  identified,  individuals  can  establish  goals 
and  associated  reward  contingencies  to  energize  and  direct  necessary  behaviors  (Mahoney 
&  Arnkoff,  1978,  1979;  Manz  &  Sims,  1980;  Neck  &  Manz,  2007).  Additionally,  self- 
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correcting  feedback,  consisting  of  a  positiveiy  framed  reflection  on  failures  and  undesirabie 
behaviors,  may  be  quite  effective  in  helping  to  recast  these  behaviors  in  more  positive 
directions  (Manz  &  Sims,  2001). 

Natural  Reward  Strategies  inciude  the  ability  of  the  individuai  to  find  pleasure  in  the 
work  that  has  to  be  performed  and  to  focus  on  the  inherently  enjoyable  aspects  of  task  or 
activity,  leading  to  increased  feeiings  of  competence,  seif-control  and  a  sense  of  purpose 
(Csikszentmihalyi,  1996;  Deci  &  Ryan,  1985;  Herzberg,  Mausner  &  SnydermanI,  2003). 
Natural  reward  strategies  include  building  more  pleasant  and  enjoyable  features  into  a  task 
or  activity  so  that  the  task  itself  becomes  more  intrinsically  rewarding,  and  shifting  mental 
focus  to  inherently  rewarding  aspects  of  the  task  (Neck  &  Houghton,  2006;  Neck  &  Manz, 
2007). 

Constructive  Thought  Strategies  focus  on  directing  and  reshaping  various  mental 
processes — including  beliefs  and  assumptions,  self-verbalizations  (self-talk),  and  mental 
imagery — in  order  to  create  constructive  thought  patterns  and  habituai  ways  of  thinking  that 
may  have  a  positive  impact  on  individual  performance  (Neck  &  Houghton,  2006;  Neck  & 
Manz,  1992,  1996).  For  example,  individuals  can  assess  their  thought  patterns  in  an  effort 
to  identify  and  eliminate  dysfunctional  beliefs  and  assumptions  with  more  rational  and 
constructive  ones  (Burns,  1980;  Ellis,  1977;  Neck  &  Manz,  1992).  Similarly,  self-talk, 
defined  as  what  we  covertly  tell  ourselves,  can  be  closely  examined  in  order  to  eliminate 
undue  negativity  and  pessimism.  Research  in  various  fields  (sports  psychology,  clinical 
psychology,  education  and  communication)  supports  the  use  of  positive  self-talk  as  an 
effective  way  to  improve  individual  performance  (e.g..  Neck  &  Manz,  1992).  Mental  imagery 
invoives  symboiicaiiy  experiencing  behaviorai  outcomes  prior  to  actuai  performance  without 
overt  physicai  muscular  movement  (Driskell,  Copper,  &  Moran,  1994;  Finke,  1989;  Neck  & 
Manz,  1992,  1996).  Research  suggests  that  people  who  visualize  successful  performance 
before  actually  engaging  in  performance  are  much  more  likely  to  perform  successfully  when 
faced  with  the  actual  task  (Neck  &  Houghton,  2006).  In  a  meta-analysis  of  35  empiricai 
studies,  Driskell  et  al.  (1994)  reported  an  overall  positive  and  significant  effect  for  mental 
imagery  on  individual  performance. 

Theorists  have  often  suggested  a  reiationship  exists  between  self-ieadership  and 
creativity  (e.g.,  DiLiello  &  Houghton,  2006;  Houghton  &  Yoho,  2005;  Manz  &  Sims,  2001). 
The  relationship  between  creativity  and  self-leadership  may  be  partially  founded  on  the 
concepts  of  autonomy  and  self-determination.  Autonomy,  a  key  aspect  of  creativity  (e.g., 
Amabile,  1996;  Barron  &  Harrington,  1981;  Woodman  et  ai.,  1993),  has  been  linked  to  seif- 
determination  and  intrinsic  motivation  (Deci  &  Ryan,  1985).  Self-determination  is  a  primary 
component  of  self-leadership’s  natural  reward  strategies  (Neck  &  Manz,  2007).  Indeed, 
empirical  research  suggests  that  an  individual’s  need  for  autonomy  can  subsequently 
influence  the  extent  to  which  the  individual  engages  in  self-leadership  (Yun,  Cox,  &  Sims, 
2006). 

Other  reiationships  between  creativity  and  seif-leadership  have  also  been  suggested. 
For  example,  Houghton  and  Yoho  (2005)  have  suggested  a  relationship  between  individuai 
self-leadership  and  subsequent  levels  of  individual  independence  and  creativity.  In  addition, 
internai  iocus  of  controi,  a  theorized  component  of  creativity,  has  been  empirically  related  to 
individual  self-leadership  (Kazan  &  Earnest,  2000).  Finally,  an  empowering  leadership  styie 
(leading  others  to  be  self-leaders)  tends  to  promote  creativity  rather  than  conformity  (Manz  & 
Sims,  2001 ).  Indeed,  creativity  may  be  one  of  the  most  essential  aspects  of  effective 
organizational  leadership  (Mumford  &  Conneily,  1999).  Creative  thinking  and  a  different 
style  of  leadership  are  necessary  to  provide  flexibility,  facilitate  change  and  redesign 
traditional  bureaucratic  processes  (Katz  &  Kahn,  1978).  Encouraging  self-leadership  is  a 
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relatively  new  leadership  styie  that  may  help  to  promote  an  organizational  climate  that 
supports  creativity.  Empowering  ieadership  is  rapidiy  becoming  a  key  success  strategy  in 
the  rapidiy  changing  work  environments  of  the  21  century. 

The  purpose  of  the  current  study  is  to  examine  the  reiationships  between  self¬ 
leadership  and  creativity  in  the  context  of  a  defense  acquisition  organization.  More 
specificaiiy,  we  wiil  examine  possible  differences  in  seif-ieadership,  creativity  and  perceived 
organizational  support  for  creativity  between  iine  and  supervisory  defense  acquisition 
empioyees.  The  present  study  will  contribute  to  the  seif-leadership  and  creativity  iiterature 
in  a  number  of  important  ways.  First,  this  study  will  take  an  empirical  step  toward 
understanding  the  nature  of  the  reiationship  between  self-ieadership  and  creativity.  This 
study  will  also  examine  the  role  of  organizational  support  in  faciiitating  practiced  creativity 
among  organizational  members.  Most  importantly,  this  study  is  among  the  first  to  examine 
differences  in  self-leadership,  creativity  and  perceptions  of  support  between  line  and 
supervisory  empioyees.  Understanding  these  differences  may  be  a  criticai  for  reducing  the 
gap  between  creative  potential  and  practiced  creativity  in  organizations.  Finaiiy,  this  study 
makes  a  unique  contribution  to  our  knowledge  of  creativity  and  seif-ieadership  in  the  context 
of  defense  acquisitions.  The  differences  examined  here  may  have  important  implications  for 
creating  a  defense  acquisitions  workforce  with  strong  seif-ieaders  working  in  environments 
that  support  creativity.  Creative  self-ieaders  couid  synergistically  assist  the  DoD  in 
maintaining  an  aii-important  competitive  advantage  in  the  face  of  a  wide  range  of  21  ®‘- 
century  chalienges. 

Method 

Sample  and  Procedure 

Primary  data  were  coiiected  from  the  Army  Contracting  Agency  (ACA)  as  part  of  a 
iarger  study  that  examined  a  number  of  performance-reiated  issues.  Approximateiy  37%  of 
the  total  ACA  workforce  of  approximately  1900  people  chose  to  complete  the  online  survey, 
a  fairly  high  response  rate  when  compared  to  the  response  rates  for  other  federal  employee 
surveys  and  to  response  rates  for  e-mail  surveys  in  general  (Sheehan,  2001).  Listwise 
deletion  for  missing  data  resulted  in  a  final  overall  sample  of  654.  This  sample  was 
subsequently  divided  into  two  subsamples  (i.e.,  supervisory  employees,  N=215;  and  line 
employees,  N=439)  for  further  analysis.  The  average  age  of  the  respondents  was 
approximately  46,  and  the  average  job  tenure  was  approximately  12  years.  Sixty  percent  of 
the  respondents  were  female.  The  online  survey  was  activated  in  accordance  with  the 
tailored  design  method  (Dillman,  2000).  An  initial  e-mail  was  sent  to  ACA  workforce 
members  that  included  an  Informed  Consent  Notification,  the  purpose  of  the  study,  the 
approval  and  sponsorship  of  the  study,  a  confidentiality  statement  and  a  link  to  the  online 
survey.  A  subsequent  follow-up  e-mail  summarized  the  first  message,  added  a  personal 
note  and  provided  a  four-day  extension,  along  with  a  link  to  the  online  survey. 

Measures 

Thirteen  items  from  the  Revised  Self-leadership  Questionnaire  (Houghton  &  Neck, 
2002)  were  used  to  measure  self-leadership.  Twelve  items  were  utilized  to  measure 
creativity:  six  items  assessing  creative  potential  and  six  items  representing  practiced 
creativity.  These  items  have  demonstrated  fairly  good  reliability  and  validity  for  measuring 
creative  potential  and  practiced  creativity  (DiLiello  &  Houghton,  2007).  Perceived 
organizational  support  was  measured  with  six  items  from  “Keys:  Assessing  the  Climate  for 
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Creativity,”  used  with  the  permission  of  the  Center  for  Creative  Leadership  (Amabile  et  al., 
1999).  All  items  were  measured  utilizing  a  five-point  Likert-type  scale  ranging  from  Strongly 
Agree  to  Strongly  Disagree. 

Analyses 

Mean  differences  between  supervisory  and  line  employees  for  self-leadership, 
creative  potential,  practiced  creativity,  a  gap  score  (i.e.,  the  difference  between  creative 
potential  and  practiced  creativity  that  represents  untapped  creative  potential),  and 
perceptions  of  organizational  support  for  creativity  were  examined  using  a  series  of 
t-tests.  In  addition,  a  series  of  regression  analyses  were  conducted  to  examine  the  effects  of 
self-leadership,  perceived  organizational  support  for  creativity  and  organizational  level  (line 
vs.  supervisory)  on  creative  potential,  practiced  creativity  and  gap  scores,  respectively, 
along  with  the  effects  of  organizational  level  (line  vs.  supervisory)  on  perceived 
organizational  support  for  creativity. 

Results 

Means  and  standard  deviations  for  both  supervisory  and  line  employees  for  self¬ 
leadership,  creative  potential,  practiced  creativity,  gap  scores  and  perceived  organizational 
support  for  creativity  are  shown  in  Table  1 .  The  analysis  indicated  no  mean  difference 
between  groups  for  self-leadership,  f(507)  =  1 .16,  p  =  .247.  In  contrast,  analyses  showed 
significant  mean  differences  between  the  two  groups  for  creative  potential,  f(652)  =  3.30,  p  = 
.001;  practiced  creativity,  f(469)  =  7.48,  p  =  .000;  gap  scores,  f(471)  =  -5.03,  p  =  .000;  and 
perceived  organizational  support  for  creativity,  f(652)  =  3.21 ,  p  =  .001 . 

Table  1.  Means  and  Standard  Deviations  (in  parentheses) 


SL 

CP 

PC 

GS 

OS 

Supervisors 

49.55 

25.47 

23.58 

1.89 

20.00 

N  =  2^5 

(6.10) 

(2.98) 

(4.04) 

(4.10) 

(5.40) 

Line  Empioyees 

48.92 

24.65 

20.97 

3.68 

18.54 

W  =  439 

(7.43) 

(3.03) 

(4.51) 

(4.60) 

(5.46) 

Note:  SL=Self-Leadership,  CP=Creative  Potential,  PC=Practiced  Creativity,  GS=Gap  Score, 
OS=Perceived  Organizational  Support. 


Four  separate  regression  analyses  were  conducted.  Model  1  examined  the  effects 
of  the  independent  variables  self-leadership  and  organizational  level  (1=supervisor  -  0=line, 
using  dummy  variable  coding)  on  the  dependent  variable  creative  potential.  Model  2 
examined  the  effects  of  self-leadership,  perceived  organizational  support  for  creativity,  and 
organizational  level  on  the  dependent  variable  practiced  creativity.  Model  3  examined  the 
relationships  between  the  three  independent  variables  and  gap  scores.  Finally,  Model  4 
explored  the  effects  of  organizational  level  on  perceptions  of  organizational  support  for 
creativity.  A  summary  of  the  results  of  these  analyses  is  presented  in  Table  2. 


The  regression  equation  for  Model  1  suggested  that  both  self-leadership  and 
organizational  level  were  significantly  related  to  creative  potential,  with  self-leadership  as  the 
stronger  predictor  of  the  two  (Standardized  |3  =  .356,  p  =  .000).  The  equation  for  Model  2 
indicated  that  self-leadership,  perceived  organizational  support  for  creativity  and 
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organizational  level  were  all  significant  predictors  of  practiced  creativity,  accounting  for 
approximately  42.6%  of  its  variance.  Of  the  three  variables,  perceived  organizational 
support  was  the  stronger  predictor  of  practiced  creativity  (Standardized  p  =  .563,  p  =  .000). 
The  Model  3  analysis  found  that  perceived  organizational  support  and  organizational  level 
were  significantly  and  negatively  related  to  gap  scores,  explaining  approximately  33.1%  of 
the  observed  variance.  The  regression  equation  suggested  a  strong  negative  effect  for 
perceived  organizational  support  (Standardized  p  =  -.551 ,  p  =  .000),  indicating  that  lower 
perceptions  of  organizational  support  for  creativity  will  result  in  larger  gaps  between 
individuals’  creative  potential  and  their  practiced  creativity.  In  addition,  the  equation 
suggests  that  gap  scores  will  be  significantly  greater  for  line  employees  than  for  supervisors 
(Organizational  Level:  Standardized  p  =  -.1 17,  p  =  .000).  Finally,  the  regression  analysis  for 
Model  4  implied  that  supervisors  tend  to  have  more  positive  perceptions  of  organizational 
support  for  creativity  than  line  employees 

(Organizational  Level:  Standardized  p  =  .125,  p  =  .001 

Table  2.  Summary  of  Regression  Analyses  Results 


Independent 

Variables 

Model  1; 

P 

Creative 
Potential 
p  -  value 

Model  2: 

P 

Praetieed 
Creativity 
p  -  value 

Model  3: 

P 

Gap  Seore 
p  -  value 

Model  4: 

p 

Organizati 

onal 

Support 
p  -  value 

Self-leadership 

.356 

.000 

.158 

.000 

Pereeived 

Organizational 

Support 

.563 

.000 

-.551 

.000 

Organizational 

Level 

.113 

.002 

.195 

.000 

-.117 

.000 

.125 

.001 

Adjusted 

.140 

.426 

.331 

.014 

F  Statistie 

54.25 

162.84 

162.53 

10.32 

p  -  value 

.000 

.000 

.000 

.001 

Discussion 

This  study  revealed  a  number  of  significant  differences  between  line  and  supervisory 
acquisition  employees.  Our  analyses  suggested  that  self-leadership  was  significantly 
related  to  creative  potential  and  practiced  creativity  for  both  line  and  supervisory  employees, 
with  no  significant  differences  in  overall  levels  of  self-leadership  between  the  two  groups.  In 
contrast,  we  found  significant  differences  between  line  and  supervisory  employees  in 
creative  potential,  practiced  creativity,  gap  scores  and  perceptions  of  organizational  support 
for  creativity.  Specifically,  line  employees  reported  significantly  lower  levels  of  creative 
potential,  practiced  creativity  and  perceptions  of  organizational  support  for  creativity,  along 
with  higher  gap  scores  in  comparison  to  supervisors. 

Our  analyses  further  suggested  that  although  supervisors  tend  to  have  more  creative 
potential  than  line  employees,  self-leadership  appears  to  be  the  more  important  concept  in 
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determining  an  individual’s  creative  potential.  Likewise,  although  self-leadership  and 
organizational  level  are  both  important  determinants  of  practiced  creativity,  employee 
perceptions  of  organizational  support  for  creativity  seem  to  be  far  more  crucial.  Similarly, 
perceived  organizational  support  for  creativity  appears  to  be  more  important  than 
organizational  level  in  predicting  creativity  gaps  in  acquisition  employees.  In  other  words, 
employees  who  feel  that  the  organization  supports  their  creative  efforts  will  be  much  more 
likely  to  practice  creative  behaviors,  thus,  lowering  the  gap  between  their  potential  and 
practiced  creativity.  Finally,  organizational  level  was  a  significant  determinant  of  perceptions 
of  organizational  support  for  creativity — with  supervisory  employees  holding  significantly 
more  positive  perceptions  of  support  than  line  employees.  In  summation,  our  analyses 
suggest  that  self-leadership  may  be  a  key  determinant  of  creative  potential  and  practice 
among  defense  acquisition  employees  and  that  perceptions  of  organizational  support  for 
creativity,  which  tend  to  be  weaker  in  non-supervisory  employees,  are  critical  In  determining 
whether  creative  potential  will  be  realized  or  whether  a  gap  between  potential  and  practice 
will  result. 

The  results  of  this  study  have  important  theoretical,  empirical  and  practical 
applications.  This  study  adds  to  our  understanding  of  the  nature  of  the  relationship  between 
self-leadership,  creativity  and  organizational  support  for  creative  practices  at  both  the 
supervisory  and  non-supervisory  levels.  Our  findings  imply  that  self-leadership  is  a  primary 
tool  for  facilitating  creativity  at  all  organizational  levels  and  that  active  organizational  support 
for  creativity  may  be  the  key  for  reducing  the  gap  between  creative  potential  and  practiced 
creativity  that  represents  untapped  creative  resources.  Our  results  also  suggest  that  this 
gap  is  much  more  pronounced  among  line  employees  and  that  line  employees  generally 
perceive  less  organizational  support  for  utilizing  their  creative  resources  than  supervisors. 

In  order  to  address  this  situation,  an  organizational  intervention  designed  to  increase  self¬ 
leadership  capabilities  at  all  levels  and  to  increase  perceptions  of  organization  support  for 
creative  practices  among  line  employees  in  defense  acquisitions  would  be  well  advised. 
More  specifically,  a  structured  self-leadership  training  program  similar  to  those  reported 
elsewhere  in  the  literature  (e.g..  Neck  &  Manz,  1996;  Stewart,  Carson,  &  Cardy,  1996)  could 
be  conducted  for  acquisition  employees.  Such  a  training  program  could  have  the  dual  effect 
of  increasing  self-leading  behaviors  and,  thus,  creative  potential  while  also  strongly  signaling 
organizational  support  for  creative  behaviors. 

Although  our  findings  suggest  exciting  avenues  toward  increasing  self-leadership 
and  unleashing  creative  resources  at  all  organizational  levels,  our  study  is  bound  by  certain 
limitations.  First,  the  present  sample  was  relatively  homogeneous,  consisting  entirely  of 
members  of  the  Army  Contracting  Agency.  As  we  have  suggested,  such  a  sample  is 
especially  appropriate  for  creativity  research  because  the  Department  of  Defense  has  taken 
a  keen  interest  in  tapping  all  creative  resources  available  in  order  to  sustain  a  competitive 
advantage.  However,  it  is  uncertain  as  to  whether  the  results  reported  here  would 
generalize  to  other  samples  of  Interest.  Second,  all  items  were  self-reported  and  collected 
utilizing  a  single  survey  at  a  single  point  in  time,  thus  raising  concerns  regarding 
measurement  issues  such  as  response  set  and  social  desirability  biases.  Given  these 
potential  problems,  our  findings  should  be  viewed  with  some  degree  of  caution.  On  the 
other  hand,  despite  such  inherent  limitations,  the  use  of  self-reported  items  collected  in  a 
single  administration  is  common  practice  in  many  aspects  of  social  science  research. 

Future  research  should  continue  to  examine  the  relationships  between  self¬ 
leadership,  creative  potential,  practiced  creativity,  organizational  level  and  organizational 
support  for  creativity.  Specifically,  future  research  should  more  closely  examine  the  role  of 
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organizational  support  as  a  moderator  of  the  relationship  between  creative  potential  and 
practiced  creativity  and  as  a  key  mechanism  for  reducing  the  gap  between  these  concepts 
in  organizations.  In  addition,  perceptions  of  support  for  creativity  might  be  further 
subdivided  from  the  organizational  level  to  the  work  group  and  supervisory  levels  in  order  to 
provide  additional  insights  (DiLiello  &  Houghton,  2006).  Similarly,  future  research  could 
continue  to  examine  the  differences  between  line  and  supervisory  employees  in  terms  of 
creativity  and  perceptions  of  support  for  creative  practices,  with  an  eye  toward  identifying 
ways  to  increase  creativity  at  all  organizational  levels.  In  closing,  our  findings  and 
suggestions  have  significant  practical  application  in  the  context  of  the  transformational 
efforts  in  the  Department  of  Defense  in  support  of  warfighter  readiness.  An  acquisition 
workforce  of  creative  self-leaders  could  synergistically  assist  the  organization  in  maximizing 
the  utility  of  all  organizational  resources. 
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Abstract 

The  Tuckman  (1965)  four-stage  sequential  model  of  team  development  (Forming, 
Storming,  Norming,  and  Performing,  or  FSNP)  represents  today's  most  widely  used  model. 
However,  the  Tuckman  model  is  a  conceptual  statement  that  was  suggested  by  the  data 
and  has  not  been  empirically  validated  (Tuckman,  1965).  Hadyn,  Teare,  Scheuing  and 
Armistead  (1997,  p.  118)  state  that,  "despite  increasing  interest  in  teamwork,  much  of  the 
literature  on  the  subject  is  inconclusive  and  often  derived  from  anecdote  rather  than  primary 
research." 

The  goal  of  this  research  was  to  develop  empirical  evidence  to  determine  whether  or 
not  the  Tuckman  model  or  some  variant  thereof  provides  an  appropriate  model  to  explain 
the  development  of  small,  short-duration  technical  teams  within  the  Acquisition  Community. 

The  results  showed,  to  a  95%  confidence  level,  that  only  about  2%  of  321  teams 
studies  followed  the  Tuckman  model  (FSNP).  However  a  modified  model,  called  the  DAU 
Model  (FNP — Tuckman  model  sans  Storming),  was  experienced  by  229  of  the  321  teams 
(77%).  This  discrete  three-stage  model,  along  with  a  redefined  Storming  function  that  takes 
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place  throughout  the  teams'  duration,  constitutes  a  strong  model  of  team  dynamics  for  the 
studied  Acquisition  population. 


This  research  demonstrates  that  not  only  do  technical  teams  generally  follow  the 
DAU  model,  but  also  that  there  is  a  strong  correlation  between  teams  producing  above- 
average  products  and  teams  following  this  model.  The  results  of  this  research  strongly 
suggest  the  possibility  that  the  productivity  of  a  technical  team  may  be  significantly  improved 
by  guiding  its  development  through  a  well-defined  process. 

Background 

How  to  build  effective  teams  is  one  of  the  most  important  management  issues  of  the 
day.  Significant  effort  is  being  expended  to  gain  a  better  understanding  of  how  highly 
successful  teams  develop  in  hopes  that  methodologies  to  enhance  team  productivity  can  be 
produced  that  will  accelerate  the  movement  of  high-quality  products  to  the  marketplace 
(Osterman,  1994).  In  Quality  Circles,  Concurrent  Engineering,  and  many  other 
management  innovations,  the  team  is  the  organizational  unit  to  which  creative  control  is 
being  delegated.  As  a  result,  there  is  a  great  need  to  better  understand  the  development  of 
technical  teams. 

The  culture  of  many  of  today’s  businesses  places  as  much  emphasis  on  a  person’s 
ability  to  work  together  effectively  in  a  team  environment  as  on  technical  skills  (Tarricone, 
2002).  Osterman  (1994)  found  that  teams  are  being  used  extensively  by  organizations  that 
need  to  get  products  to  market  faster.  Some  industries  have  reported  that  teaming  brings 
advantages  such  as  increased  productivity  and  decreased  absenteeism  (Beyerlein,  2001). 
According  to  Beyerlein,  the  use  of  task-oriented  teams  within  organizations  has  spread 
across  many  industries,  nonprofits,  and  national  boundaries  in  the  last  decade.  KInlaw 
(1991)  found  that  teamwork  is  the  main  driver  for  continuous  improvement  and  increased 
competitiveness.  According  to  Marks  (2001),  the  advantage  of  teamwork  is  that  people 
working  together  can  often  achieve  something  beyond  the  capabilities  of  individuals  working 
alone.  Furthermore,  Marks  points  out  that  success  is  not  only  a  function  of  team  members' 
talents  and  the  available  resources  but  also  of  the  processes  team  members  use  to  interact 
with  each  other.  Research  on  the  development  and  functioning  of  teams  is  needed  to 
enable  organizations  to  retool  human  resource  systems  so  that  managers  can  better  select, 
train,  develop,  and  reward  personnel  for  effective  teamwork  (Marks,  2001).  To  remain 
competitive,  it  is  important  for  organizations  to  understand  how  to  create  and  maintain 
teams  that  are  highly  effective  in  today’s  globally  competitive  environment  (Yancey,  1998). 

Introduction — the  Importance  of  Teams  to  Defense  Acquisition 
and  the  Defense  Acquisition  University  (DAU)  Connection 

Short-duration,  small  technical  teams  represent  a  significant  proportion  of  the  team 
activities  within  the  Department  of  Defense  (DoD)  acquisition  community  and  corporate 
organizations.  These  teams  come  together,  focus  on  the  task  at  hand,  produce  whatever 
products  are  required,  communicate  their  results,  and  then  disband  as  easily  and  quickly  as 
they  were  formed  (Canadian  Business,  2001).  Wherever  highly  specialized  knowledge 
spanning  multiple  disciplines  is  required,  the  technical  team  enjoys  widespread  use.  Some 
examples  are  as  follows: 

■  Multi-disciplinary  Product  Integration  Teams 
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■  Tiger  Teams  (narrow  focus,  single  issue) 

■  Proposal  Teams 

■  Design  Teams 

■  Educational/Training  Teams 

■  Problem  Resolution  Teams 

■  Product  Development  Teams 

■  Marketing/Sales  Teams 

In  today’s  environment,  short-duration,  small  technical  teams  drive  an  enormous 
quantity  of  critically  important  decisions  within  a  broad  range  of  organizations  in  all  sectors 
of  the  US  economy.  The  DoD  acquisition  community  is  one  such  sector  that  makes 
extensive  use  of  technical  teams.  Thus,  understanding  how  these  teams  develop  is  of 
critical  importance  to  the  DoD  Acquisition  Community. 

DoD  acquisition  professionals  are  those  in  the  government  who  are  responsible  for 
acquiring  weapon  systems  for  the  Department  of  Defense.  Their  collective  decisions,  made 
primarily  by  technical  teams,  move  hundreds  of  billions  of  dollars  per  year,  influence  the 
outcome  of  international  conflicts,  and  determine  the  effectiveness  of  the  US  military.  To 
perform  its  mission,  the  acquisition  community  employs  thousands  of  technical  teams  to 
develop  the  information  necessary  to  make  critical  decisions  and  to  integrate  the 
development  and  production  of  very  large,  costly,  and  complex  weapon  systems.  The 
Integrated  Product  Teams  (IPTs),  which  has  been  organic  to  both  industry  and  DoD 
acquisition  for  many  years,  is  a  good  example  of  a  technical  team.  The  IPT,  along  with  all  of 
the  short-duration,  small  sub-teams  it  spawns  is  increasingly  being  hailed  as  the  preferred 
way  to  manage  large-scale  acquisitions  (Weinstock,  2002,  p.  1 ).  DoD  Directive  5000. 1 
requires  that  the,  “Acquisition  Community  implement  the  concept  of  Integrated  Product  and 
Process  Development  (IPPD)  utilizing  IPTs  as  extensively  as  possible”  (DAU,  2004,  October 
17,  p.  113). 

DoD  technical  teams  are  often  multi-disciplinary  and  could  include  scientists  and 
engineers  as  well  as  management,  contracts,  budget,  security,  quality,  survivability,  and 
logistics  personnel  from  both  the  developer  and  the  user  organizations  (DAU,  2004,  October 
17).  DoD  teams  often  include  contractor  personnel  as  well  as  government  employees.  DoD 
acquisition  activity  centers  on  extremely  large  and  complex  systems  that  often  push  the 
state-of-the-art  in  many  fields  simultaneously.  The  acquisition  workforce  numbers 
approximately  133,000  people,  including  both  military  and  civilians.  It  is  vital  to  the  success 
of  integrated  military  systems  that  all  the  stakeholders  work  together  as  efficiently  and 
productively  as  possible  (Weinstock,  2002,  August  15). 

Because  countless  lives,  billions  of  dollars  and  the  national  interest  are  at  stake,  the 
US  Congress  required  the  Department  of  Defense  to  take  action  to  promote  high  levels  of 
professionalism  and  competency  within  its  acquisition  workforce.  One  action  taken  by  the 
DoD  was  to  establish  a  process  of  training  and  certification  for  individuals  in  the  acquisition 
workforce.  The  Defense  Acquisition  University  (DAU)  was  established  to  implement  this 
training.  This  process,  called  the  Acquisition  Certification  Program,  was  designed  to  ensure 
that  an  employee  meets  the  professional  standards  (education,  training  and  experience) 
established  for  acquisition  career  positions  at  three  separate  levels  of  decision-making 
responsibility;  in  addition,  promotion  opportunities  are  tied  to  these  certification  levels. 
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The  DAU  charter  is  to  provide  training  to  the  DoD  workforce  that  sets  the  direction  for 
all  DoD  acquisitions.  Due  to  the  emphasis  the  DoD  places  on  teamwork,  many  of  the  DAU 
classes  are  conducted  utilizing  student  teams  to  generate  typical  DoD  acquisition  products. 
Examples  of  classes  that  make  use  of  teams  are:  Systems  Engineering,  Program 
Management,  Software  Acquisition  Management,  and  Information  Technology  Acquisition 
Management.  The  DAU’s  use  of  student  teams  is  consistent  with  many  conventional 
universities  who  are  also  requiring  teaming  activities  in  their  courses.  These  student  teams 
are  used  to  enable  the  generation  of  more  complex  products  and  to  prepare  the  students  for 
the  inevitable  teaming  requirement  in  the  workforce.  It  was  these  DAU  teams  that  were 
studied  by  this  research. 

The  Tuckman  Model 

In  1965,  Tuckman  examined  50  empirical  research  efforts  to  arrive  at  his  own  group 
dynamics  model.  Tuckman  (1965)  concluded  that  groups  develop  through  a  sequence  of 
four  discrete  stages:  the  first  stage.  Forming,  is  the  initial  group  coming  together;  the 
second  stage,  Storming,  involves  conflict  among  the  group  members;  the  third 
stage,  Norming,  is  when  the  group  actually  begins  to  find  value  in  working  together 
and  establishes  processes  that  enable  the  group  to  function;  and  the  fourth  stage. 
Performing,  represents  the  time  when  the  group  is  working  together  smoothly  and  is 
able  to  share  ideas  and  accomplish  goals.  However,  Tuckman  (1965)  warned  researchers 
that  the  application  of  this  model  to  generic  team  settings  may  be  inappropriate  since  the 
majority  of  his  data  came  from  the  population  of  therapy  groups  and  human  relations 
training  groups.  Note  that  the  types  of  groups  from  which  the  Tuckman  model  was  derived 
have  almost  nothing  in  common  with  the  technical  groups  supporting  DoD  acquisition. 

Many  government  organizations,  contractors,  and  management  consultants  appear 
to  be  working  under  the  assumption  that  a  team’s  productivity  can  be  significantly  improved 
by  optimally  guiding  the  interaction  of  the  team’s  members  through  the  Tuckman  model’s 
sequence  of  stages  (Glacel  &  Robert,  1995).  Buchanan  and  Huczynski  (1997)  found  the 
Tuckman  model  to  be  the  preferred  model  of  team  development  for  all  types  of  teams.  It  Is 
widely  believed  in  both  industry  and  government  that  a  leadership  knowledgeable  in  how  to 
apply  Tuckman’s  theory  of  team  dynamics  can  markedly  enhance  teaming  performance. 
Top-tier  consulting  firms  are  teaching  or  offering  training  services  based  at  least  partially 
upon  the  assumption  that  the  Tuckman  model  applies  generically  to  most  teaming 
arrangements  (Glacel  &  Robert,  1995;  Smith,  2005).  Many  DoD  organizations  have 
received  such  training.  Glacel  and  Robert  (1995)  state  that  the  Tuckman  model  can  be 
used  to  facilitate  any  team-development  process.  They  present  the  efficacy  of  the  Tuckman 
model  as  a  general  model  that  applies  to  all  teams.  They  state  with  certainty:  “In  the 
development  of  any  team,  certain  stages  of  behavior  [Tuckman  stages  model]  take  place 
which  impact  how  well  the  individuals  and  the  team  accomplish  their  task”  (Glacel  &  Robert, 
1995,  p.  97). 

Notwithstanding  its  widespread  use,  Tuckman  did  not  empirically  validate  his  model 
(Tuckman,  1977).  The  government  and  industry  managers  are,  thus,  teaching  and 
implementing  a  team-development  model  that  has  never  be  validated  for  any  type  of  team, 
including  the  technical  teams  that  are  predominant  within  the  DoD  acquisition  process. 

Large  sums  of  money  and  critical  outcomes  may  be  influenced  by  the  wide  use  of  the 
Tuckman  Theory,  which  was  primarily  developed  through  an  analysis  of  data  describing  the 
development  of  therapy  groups  and  human  relations  training  groups  during  the  mid-1960’s. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  -  75  - 


Tuckman  himself  warned  the  group  development  community  that  his  stage  model 
had  never  been  empirically  validated  and  recommended  caution  in  applying  it  to  other 
settings  (Tuckman,  1965).  Subsequent  to  the  original  work,  Tuckman  and  Jensen  (1977) 
reviewed  another  22  studies  in  an  effort  to  determine  if  anyone  had  validated  the  Tuckman 
model.  In  1977,  the  only  new  research  that  had  attempted  to  validate  the  model  was  Runkel 
(1971).  Runkel  partially  supported  the  Tuckman  model;  however,  Tuckman  and  Jensen 
(1977)  felt  that  the  results  were  not  necessarily  reliable  due  to  the  researcher’s 
methodology. 

Even  if  the  Tuckman  model  of  group  development  was  valid  for  therapy  groups  and 
human  relations  training  groups,  there  is  no  reason  to  assume  that  it  would  be  applicable  to 
groups  in  other  settings.  Do  the  members  of  a  missile  design  team  interact  in  the  same  way 
as  the  members  of  a  psychiatric  therapy  group?  Perhaps,  but  independent  empirical 
validation  is  needed  before  giving  credibility  to  such  an  assumption. 

Data  Collection 

The  objective  of  this  research  was  to  establish  and  execute  a  methodology  that 
would  enable  an  objective,  rigorous  analysis  of  a  large  number  of  teams  in  order  to 
determine  whether  these  teams  were  following  the  four-stage  Tuckman  model,  or  some 
variant  thereof.  For  this  research,  the  team  members  were  drawn  from  the  population  of 
students  attending  the  DoD  Defense  Acquisition  University  (DAU)  courses.  The  DAU 
employs  technical  acquisition  teams  in  most  of  its  classroom  courses  to  emulate  the 
activities  that  acquisition  professionals  face  in  their  everyday  work  experiences.  The 
classroom  courses  are  used  to  provide  hands-on  experiential  learning.  Experiential  learning 
at  DAU  requires  that  students  work  in  teams  in  which  they  gain  professional  experience 
solving  real-world  problems  that  closely  mirror  both  the  teams  and  the  tasks  that  they 
encounter  in  their  workplace  environment. 

These  DAU  teams  could  technically  be  classified  as  academic  teams  because  they 
take  place  in  a  classroom  where  an  instructor  assigns  the  team  project.  However, 
functionally  it  could  be  argued  that  they  are  more  like  work  teams  because  the  assigned 
tasks  emulate  real-world  problems  that  the  team  members  are  typically  asked  to  solve  in  a 
work-team  environment  within  their  own  organizations.  The  DAU  teams  are  brought 
together  to  learn  and  to  practice  working  real-world  problems.  If  the  DAU  team  members 
are  role  playing,  then  the  role  they  are  playing  is  themselves  at  work. 

As  is  the  case  with  work  teams,  the  researcher  had  no  control  over  the  team  tasks. 
Individual  team  projects,  which  take  from  one  to  twenty  hours  of  team  interaction  to 
complete,  are  relevant  to  the  tasks  team  members  accomplish  within  their  own 
organizations.  The  team  projects  are  selected  by  the  course  instructor.  DAU  teams 
normally  contain  4  to  8  team  members. 

All  team  exercises  within  the  DAU  require  products  to  be  developed  and  delivered  by 
the  end  of  the  exercise.  The  products  delivered  in  the  class  are  similar  to  products  delivered 
in  the  DoD  acquisition  environment.  For  example,  a  Systems  Engineering  class  is  required 
to  perform  a  Requirements  Analysis  Task  within  the  class  team.  These  are  the  people  who 
perform  Requirements  Analysis  Tasks  within  the  Acquisition  Workforce. 

The  instructor  graded  each  team’s  product  quality.  It  can  be  assumed  that  students 
are  generally  motivated  to  develop  the  best  products  they  are  capable  of  producing  within 
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their  teams  because  the  quality  of  their  work  is  openly  graded.  Furthermore,  passing  DAD 
courses  is  dependent  upon  the  quality  of  their  teamwork  as  well  as  the  quality  of  their  team 
products  (in  addition  to  their  final  exam  grades).  Since  passing  a  DAU  course  earns  a 
certain  level  of  certification  within  the  Acquisition  Corps,  and  since  certification  levels  are 
tied  to  career  advancement  opportunities  (DoD,  2005,  January  12),  DAU  students  generally 
take  their  teaming  activities  seriously  and  are  motivated  to  work  well  together. 

For  this  research,  the  Diane  Miller  (1997)  Group  Process  Questionnaire  (GPQ)  was 
utilized  to  collect  data  to  determine  if  events  defining  the  Tuckman  stages  took  place  within 
the  DAU  technical  acquisition  teams  being  observed.  If  the  instrument  determines  that  a 
team  member  observed  “Tuckman  events”  taking  place  within  the  team,  then  data  is 
gathered  to  define  when  these  events  occurred  and  how  long  they  lasted.  Dr.  Miller  involved 
team  dynamics  subject-matter  experts  to  generate  her  GPQ  and  then  performed  a  validation 
study  to  eliminate  questions  that  did  not  reflect  the  team  dynamics  models  of  interest.  The 
DAU  Research  Report  entitled  “Small,  Short  Duration  Technical  Team  Dynamics”  provides 
more  details  about  how  this  instrument  was  selected  (Knight,  2006).  Miller’s  questionnaire 
contains  15  questions  that  are  reflective  of  the  Tuckman  model  (Miller,  1997).  Figure  1 
provides  a  list  of  the  15  Tuckman  questions  included  in  the  GPQ. 

The  GPQ  required  10-20  minutes  to  complete.  Each  of  the  15  Tuckman  questions 
asks  the  individual  to  determine  if  an  event  (correlated  with  one  of  Tuckman’s  four  stages) 
happened  during  a  specific  teaming  exercise  and  if  so,  when  it  happened  and  how  long  it 
lasted.  The  point  at  which  the  event  occurred  and  its  duration  were  recorded  on  a  timeline 
scaled  from  1-50.  If  the  event  was  a  singular  event  that  occurred  at  one  instant  of  time  only, 
then  the  person  would  click  a  single  unit  (box)  on  the  timeline.  If  it  occurred  various  times 
with  various  durations,  the  person  would  indicate  each  occurrence  and  its  duration  by 
clicking  a  series  of  contiguous  boxes.  A  sample  timeline  is  shown  in  Figure  2. 
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Stage 

Question 

GPQ  Question 

F1 

14 

The  team  attempted  to  discover  what  was  to  be 
accomplished 

F2 

24 

Individuals  tried  to  determine  what  was  to  be 
accomplished 

F3 

31 

The  team  tried  to  determine  the  parameters  of  the  task 

S1 

1 

There  was  conflict  between  group  members 

S2 

5 

Individuals  demonstrated  resistance  towards  the 
demands  of  the  task 

S3 

16 

The  group  was  experiencing  some  friction 

S4 

20 

Group  members  became  hostile  towards  one  another 

N1 

11 

Individuals  identified  with  the  group 

N2 

23 

Group  norms  were  developed 

N3 

26 

The  team  felt  like  it  had  become  a  functioning  unit 

N4 

30 

Group  cohesion  had  developed 

P1 

3 

Solutions  were  found  which  solved  the  problem 

P2 

6 

A  unified  group  approach  was  applied  to  the  task 

P3 

21 

Constructive  attempts  were  made  to  resolve  project 
issues 

P4 

22 

Problem  solving  was  a  key  concern 

F=Forming  S=Si 

terming  N=Norming  P=Performing 

Figure  1.  Tuckman  Questions  in  the  Group  Process  Questionnaire 
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14)  The  team  Jittempte (I  to  discover  wh^it  was  to  be  accomplished  @  YES  O  UNCERTAIN  O  NO 


M  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  kl  I  I  M  I  I  kl  kl  WWWW\  M  I  I  I  I  I  I  I  klU] 

Task  Start  Midpoint  End  of  Task 

24)  Luhvidnals  tiled  to  detenimie  what  was  to  be  accomphshed  @  YES  O  UNCERTAIN  O  NO 

M  I  I  I  I  kl  I  I  I  I  I  I  I  WM  kl  I  I  WWH  I  I  I  I  I  I  I  I  kl  I  M  I  I  I  I  I  kl  I  UDI] 

Task  Start  Midpoint  End  of  Task 

31)  The  team  tided  to  detenmiie  the  parameters  of  the  task  @  YES  O  UNCERTAIN  O  NO 

I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  kkkkki  I  I  I  I  I  kl  kl  I  I  M  kl  I  I 

I - 1 - 1 - 1 - 1 

Task  Start  Midpoint  End  of  Task 


Figure  2.  Sample  Question  Timeline 

Originally,  368  teams  were  surveyed  with  a  response  rate  of  90%.  The  research 
population  consisted  of  321  teams  and  1448  individuals.  The  average  team  size  was  4  to  5 
members  but  ranged  from  2  to  8  members.  The  durations  of  these  team  projects  ranged 
from  one  hour  to  two-and-a-half  days.  This  population  contained  68%  males,  30%  females 
and  2%  who  did  not  indicate  their  gender.  Because  the  more  technical  professions 
(particularly  engineering)  are  predominately  male,  this  lopsided  gender  breakdown  is  normal 
and  expected  within  the  DAU.  The  DAU  students  studied  in  this  research  project  represent 
a  typical  set  of  DAU  students.  They  are  generally  well-educated  career  professionals 
working  in  a  predominately  technical  environment.  Figure  3  shows  the  percent  of  team 
members  versus  highest  degree  attained.  Note  that  88%  have  at  least  a  college  degree 
(BS/BA)  and  almost  40%  have  completed  graduate  degrees.  These  team  members  are 
generally  aware  and  bright  and  should  have  no  trouble  understanding  the  questions  asked 
by  the  questionnaire  or  being  able  to  relate  those  questions  to  the  events  they  witness  in 
their  teams. 


High  School 

BS/BA 

MS/MBA 

PhD 

Doctorate 

12% 

50% 

36% 

2% 

Figure  3.  DAU  Survey  Population  Education  Levels 

The  courses  offered  at  DAU  are  typically  not  taken  by  inexperienced  acquisition 
employees.  These  are  not  entry-level  courses,  but  rather  are  aimed  at  midlevel  and  senior 
professionals  who  are  actively  trying  to  advance  their  careers.  This  group  of  career-ladder 
climbers  tends  to  have  more  drive  and  energy  and  is  a  little  more  intellectually  aggressive 
than  the  typical  acquisition  employee.  The  DAU  teams  in  the  research  population  are,  on 
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the  average,  composed  of  midlevel  (1 1 -years  experience)  professionals  on  the  way  up  in 
their  organizations.  They  have  been  working  in  product-oriented  technical  teams  in  a 
professional  capacity  for  over  7  years  and  have  previously  worked  in  teams  with  one  or  two 
of  their  current  teammates.  Incredibly  enough,  over  71%  of  them  have  had  some  training  in 
the  techniques  of  productive  teaming.  Bottom  line:  These  teams  are  highly  experienced, 
motivated,  and  well  prepared  to  work  efficiently  together  to  produce  whatever  products  are 
demanded  by  their  various  class  exercises. 

The  DAU  instructor  evaluated  the  quality  of  each  team’s  products.  Figure  4  shows 
how  those  evaluations  were  distributed  over  the  321  teams.  The  instructors  judged  there  to 
be  145  above-average,  151  average  and  25  below-average  products. 


Above  Average 

Average 

Below 

Average 

Number 

145 

151 

25 

Percent 

45% 

47% 

8% 

Figure  4.  Instructor  Evaluations  of  Team  Products  for  321  Teams 


Analysis  Methodology 

This  research  defines  a  statistically  valid  teaming  experience  as  one  that  can  be 
proven  to  a  95%  level  of  confidence  to  be  derived  from  information  measured  by  the  GPQ 
that  has  been  certified  to  be  both  accurate  and  statistically  meaningful.  That  Is,  each  team’s 
qualitative  and  quantitative  experience  of  a  given  sequence  of  Tuckman  events  (as 
measured  by  the  GPQ)  must  be  shown  to  be  very  unlikely  (P<  0.05)  to  have  occurred  as  a 
result  of  random  fluctuations  in  the  data  (noise). 

An  assessment  of  the  ability  of  the  data  collection  methodology  to  fully  support  the 
goals  of  this  research  project  was  undertaken.  A  statistical  analysis  of  the  time-of- 
occurrence  data  generated  independently  by  each  DAU  team  member  clearly  demonstrated 
that  the  data  is  able  to  support  statistically  rigorous  results  and  conclusions  about  whether 
or  not  DAU  teams  followed  the  Tuckman  linear  sequential  model.  Data-quality  standards 
were  enforced  to  ensure  that  the  research  database  contained  a  minimum  of  noise  and 
disinformation.  Also,  it  was  statistically  shown  that  team  members  were  able  to  clearly 
assess  the  behavior  within  their  teams  relative  to  the  Tuckman  model  event  descriptions 
described  by  the  GPQ.  Finally,  it  was  shown  that  the  time-of-occurrence  data  upon  which 
the  results  of  this  research  are  based  contain  a  high  enough  signal-to-noise  ratio  to  ensure 
that  derived  results  can  be  scientifically  credible.  Appendices  N  and  M  in  the  DAU  research 
report  derive  the  details  supporting  these  conclusions  (Knight,  2006). 

To  show  that  each  team’s  experience  of  a  given  sequence  of  Tuckman  events  was 
very  unlikely  (P<  0.05)  to  have  occurred  as  a  result  of  random  fluctuations  in  the  data,  an 
analysis  of  the  sequences  defined  by  the  answers  to  the  questionnaire  was  undertaken. 

This  methodology  is  called  Sequence  Analysis.  The  GPQ  contains  3  Forming  questions,  4 
Storming  questions,  4  Norming  questions,  and  4  Performing  questions.  A  sample  showing 
one  quarter  of  the  sequence  analysis  algorithm  is  shown  in  Figure  5  below.  Here  we  see  the 
3  Forming  questions  (FI,  F2,  and  F3)  being  analyzed  relative  to  the  first  Storming  question 
(SI)  and  all  of  the  Norming  and  Performing  questions.  The  point  is  to  determine  the  order  in 
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which  the  four  Tuckman  stages  (F,  S,  N,  and  P)  occur  as  given  by  the  timeline  data 
associated  with  each  question.  The  timing  sequence  defining  the  Tuckman  model  is 
F<S<N<P  (the  time  when  Forming  occurs  is  earlier  than  the  time  when  Storming  occurs  is 
earlier  than  the  time  when  Norming  occurs  is  earlier  than  the  time  when  Performing  occurs). 
Similarly,  another  three  of  these  tables  are  used  for  Storming  questions  2,  3  and  4.  Possible 
responses  are  1  If  the  sequence  indicated  by  each  cell  is  followed  and  a  0  if  it  is  not.  For 
example,  in  the  data  upon  which  this  sample  is  based,  the  sequence  F1<S1<N1<P1  did 
occur.  Thus,  a  1  is  placed  in  the  appropriate  cell  (second  column,  fourth  row).  Likewise, 
since  our  data  did  not  support  the  sequence  F2<S1<N1<P2,  a  zero  is  placed  in  the  third 
column,  fifth  row.  Each  of  these  4  tables  could  produce  as  many  as  63  ones  for  a  total  of 
252  total  points  if  the  Tuckman  model  is  followed  100%  of  the  time  by  that  individual  or 
team.  These  scores  were  then  scaled  to  be  between  0  if  the  Tuckman  model  is  not  followed 
at  all  and  100  if  the  Tuckman  model  is  followed  for  all  questions. 
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Figure  5.  One  Quarter  of  Sequence  Analysis  Logical  Algorithm  (Tuckman  Filter) 

Another  factor  that  must  be  considered  to  determine  if  the  team  is  following  the 
Tuckman  model  is  how  to  combine  individual  data  into  team  data.  One  approach  would  be 
to  determine  a  team  position  on  each  Tuckman  question  and  then  run  this  team  data 
through  the  Tuckman  sequence-analysis  model.  The  other  approach  is  to  run  each 
individual’s  data  through  the  Tuckman  precedence  model  and  then  combine  the  Tuckman 
scores  for  individuals  to  come  up  with  a  team  score.  The  latter  method  was  chosen  for 
this  research.  The  reason  for  this  choice  is  that  the  alternative  requires  good  data  to  be 
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disregarded  without  good  reason  for  doing  so  other  than  to  simplify  the  calculations.  If  the 
first  approach  is  selected,  the  minority  opinion  of  the  existence  of  an  interpretative  and 
subjective  event  is  thrown  out. 

Once  a  Tuckman  score  is  determined  between  0-100,  the  significance  of  the  score 
must  be  determined.  A  Monte  Carlo  simulation  was  used  to  generate  a  reference  distribution 
of  Tuckman  scores.  A  large  number  (102,000)  of  questionnaires  were  filled  out  randomly — 
i.e.,  randomly  answering  “YES,”  “NO”  or  “UNCERTAIN”  to  each  of  the  15  Tuckman 
questions  and  then  producing  random  times-of-occurrence  for  each  “YES”  answer.  A 
Tuckman  score  was  calculated  for  each  of  the  102,000  random  teams.  A  reference 
distribution  was  generated  for  these  FSNP  scores  by  sorting  the  102,000  random  FSNP 
scores  into  100  bins.  For  example,  all  the  FSNP  scores  between  15.5  and  16.499  were 
counted,  and  that  number  was  put  into  bin  16.  Because  accuracy  improves  with  the 
number  of  samples  generated,  the  number  of  samples  used  (1 02,000)  simply  reflects  the 
practical  limits  of  the  available  computing  resources. 

Next,  integrating  over  the  distribution  produced  a  cumulative  probability  curve.  This 
probability  curve  was  then  used  to  generate  a  numerical  level  of  confidence  that  a  given 
score  was  not  produced  by  random  data.  Obviously,  very  low  FSNP  scores  requiring  little 
specific  organization  of  the  input  values  are  more  easily  produced  by  random  inputs;  yet, 
very  high  FSNP  scores  (requiring  all  F  times  to  be  less  than  all  S  times,  etc.,)  are  nearly 
impossible  to  produce  from  15  random  inputs  created  by  a  random-number  generator.  Each 
FSNP  score  produced  by  the  DAU  data  was  required  to  be  larger  than  the  random  FSNP 
score  associated  with  a  sa  =  0.05  probability  (of  being  produced  by  random  processes)  in 
order  to  be  declared  “significant.”  In  other  words,  for  an  FSNP  score  generated  by  a  DAU 
team  to  be  considered  statistically  significant,  it  must  be  large  enough  such  that  the 
probability  of  that  score  being  produced  by  random  input  data  is  less  than  0.05. 

Additionally,  a  sequence  of  consecutive  stages  must  be  composed  of  discrete, 
clearly  discernable,  separate  stages  or  it  becomes  a  mixture  of  multiple  stages — not  a 
sequence  of  stages  as  required  by  the  Tuckman  model.  If  stage  time-of-occurrences  are  so 
overlapped  and  intermingled  in  time  such  that  one  cannot  clearly  differentiate  consecutive 
stages,  then  no  bonafide  sequence  exists.  To  ensure  this  requirement  for  stage 
discreteness  was  met,  I  developed  a  stage-separation  test  that,  when  applied  to  the  data 
representing  the  experience  of  a  given  team,  would  tell  us  (to  some  statistical  level  of 
confidence)  whether  or  not  that  team’s  experience,  as  measured  by  the  GPQ,  constitutes  a 
valid  sequence  of  Tuckman  events.  In  other  words,  the  conditions  were  precisely  defined 
for  sequence  validation  that  determine  when  two  broadly  overlapping  events  belonging  to 
consecutive  stages  can  be  said  to  be  separated  in  time  such  that  they  represent  two 
discrete  and  separate  stages  to  some  specified  level  of  statistical  confidence. 

A  parametric  analysis  was  used  to  assess  the  sensitivity  of  research  results  to  the 
analytical  assumptions  driving  the  analysis  by  varying  the  thresholds  and  criteria  that 
numerically  represented  each  assumption.  User  input  parameters  specifying  constraints 
imposed  upon  the  analysis  were  established  as  user  inputs  to  the  analysis  engine  to  allow  a 
parametric  analysis  of  how  each  input  affected  both  intermediate  and  final  results. 

To  summarize:  An  individual’s  or  team’s  FSNP  score  was  counted  as  being 
supportive  of  the  Tuckman  model  only  if  its  value  was  equal  to  or  greater  than  the  calculated 
“significance  threshold”  and  if  the  FSNP  sequence  was  shown  (to  a  95%  probability)  to  have 
discrete  stages.  The  significance  threshold  is  an  FSNP  score  calculated  within  the 
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Sequence  Analysis  algorithm  associated  with  a  probability  of  0.05  that  a  given  FSNP  score 
could  have  been  generated  by  random  inputs.  From  the  random-reference  distribution  and 
its  associated  cumuiative  probability  curve,  it  was  determined  that  an  FSNP  score  of  0.0976 
had  a  probabiiity  of  0.05  of  being  random.  Thus,  any  score  equai  to,  or  greater  than,  0.0976 
represented  a  significant  score.  More  detail  on  random  Tuckman  score  distributions  and 
probability  curves  can  be  found  in  the  DAU  Research  Report  entitied  “Small,  Short  Duration 
Technicai  Team  Dynamics”  (Knight,  2006). 

In  addition  to  determining  if  an  individual  and  the  team  are  following  the  Tuckman 
model  at  the  95%  level  of  confidence,  this  research  iooked  at  what  other  possibie  forms  of 
the  Tuckman  modei  were  being  foilowed  (i.e..  Forming,  Norming,  Performing  OR  Forming, 
Norming,  Storming,  Performing,  etc.).  There  are  64  possibie  combinations  of  alternative 
sequences  of  the  Tuckman  stages.  For  each  individual  and  for  each  team,  a  calculation 
was  performed  to  determine  which  of  these  sequences  was  being  foiiowed.  This  was  then 
piotted  to  determine  which  sequences  showed  up  the  most  often.  The  two  variants  of  the 
Tuckman  sequential  stages  model  that  were  most  prevalent  were  F<N<P  and  F<N/P 
(Forming  before  Norming  and  Performing).  These  models  were  assessed  using  the  same 
anaiytical  methodoiogy.  In  the  exact  same  manner  described  above  for  creating  a 
Sequence  Analysis  aigorithm  SAf<s<n<p  that  caiculates  FSNP  scores  in  order  to  assess  the 
degree  to  which  a  statisticaily  valid  Tuckman  modei  (F<S<N<P)  was  experienced  by  DAU 
teams,  an  SAf<n<p  algorithm  was  deveioped  that  calcuiates  FNP  scores  in  order  to  assess 
the  degree  to  which  a  statisticaliy  vaiid  F<N<P  model  was  experienced  by  DAU  teams. 
Simiiarly,  an  SAf<n/p  aigorithm  was  developed  that  calculates  FN/P  scores  in  order  to 
assess  the  degree  to  which  a  statisticaiiy  valid  F<N/P  model  was  experienced  by  DAU 
teams.  The  significance  threshold  for  F<N<P  sequences  was  4.251 ,  and  the  significance 
threshold  for  F<N/P  sequences  was  6.51 1 . 

Results 

The  final  results  are  shown  in  Figure  6.  Only  6  teams  (2%)  out  of  321  experienced  a 
statisticaily  valid  Tuckman  sequence;  it  is  dear  that  the  technical  acquisition  teams  of  DAU 
did  not  follow  the  Tuckman  model.  This  outcome  was  primarily  driven  by  a  iack  of  Storming 
within  the  teams.  Secondly,  Norming  and  Performing  appear  to  be  interspersed  in  time  to 
such  an  extent  that  it  is  difficuit  to  separate  the  two. 
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Figure  6.  Results  Summary 

There  were  several  attributes  of  the  DAU  teams  that  might  possibly  be  related  to  the 
lack  of  Storming  behavior.  The  first  attribute  is  team  size.  Typical  DAU  team  sizes  were  4 
to  8  team  members.  One  might  wonder  if  small  teams  Storm  less  than  larger  teams. 

Further  research  would  have  to  be  performed  to  provide  a  conclusive  answer  to  this 
question;  however,  Benfield  (2005)  also  found  very  little  Storming  in  his  data,  yet  his  team 
sizes  were  not  restricted  to  such  small  sizes.  In  fact,  43%  of  his  teams  had  more  than  1 1 
team  members.  The  second  attribute  is  the  short  duration  of  teaming  activity.  The  median 
DAU  team  duration  was  4  hours,  while  no  team  duration  was  greater  than  20  hours.  The 
question  here  is:  Do  short-duration  teams  Storm  less  than  longer-duration  teams?  To 
conclusively  determine  the  effect  of  team  duration  upon  the  incidence  of  Storming,  further 
research  is  required.  However,  according  to  Benfield’s  (2005)  research,  53%  of  the  teams 
he  studied  lasted  longer  than  12  months  and  also  produced  very  little  Storming  behavior 
relative  to  the  other  stages. 

The  third  attribute  that  may  have  influenced  the  lack  of  Storming  within  DAU  teams  is 
team  setting.  The  DAU  teams  were  in  an  academic  setting  which,  because  of  the  nature  of 
DAU  and  DAU  teams,  could  be  considered  somewhere  between  Tuckman’s  (1965)  natural 
and  laboratory  settings;  however,  DAU  teams  are  most  similar  to  Tuckman’s  natural  teams. 
Benfield  (2005)  studied  natural  teams  working  in  a  DoD  technical  environment  and  similarly 
found  a  low  level  of  Storming  relative  to  the  other  stages.  There  is  yet  another  attribute  of 
the  DAU  academic  setting  that  may  have  influenced  the  amount  of  Storming  behavior 
exhibited.  DAU  teaming  exercises  take  place  in  the  presence  of  an  instructor  and  are 
subsequently  graded  by  this  instructor.  This  is  analogous  to  a  natural  team  when 
“management”  is  a  part  of  the  team  or  closely  monitors  the  team.  Cooperative 
professionalism  is  encouraged  while  conflict,  resistance,  and  hostility  are  often  discouraged 
whenever  a  neutral  authority  with  significant  power  over  the  team  members  is  observing  the 
process.  In  other  words,  team  members  may  have  been  exhibiting  their  best  professional 
behavior  rather  than  the  less  politically  correct  behavior  they  might  have  exhibited  within  a 
group  of  peers.  Certainly,  “resistance  to  the  task”  would  be  muted  in  the  presence  of  the 
instructor  who  assigned  the  task  and  who  was  going  to  grade  the  task  products. 

In  addition  to  the  lack  of  Storming  found,  the  distribution  of  Storming  data  was  more 
or  less  uniform  across  the  entire  timeline  (team  duration),  as  shown  in  Figure  7. 
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Figure  7.  Average  Time-of-occurrences  for  Each  Stage  for  321  Teams 

This  characteristic  of  a  constant  low  level  of  Storming  spread  evenly  across  the 
entire  duration  of  a  team’s  activity  was  also  observed  in  Benfield’s  (2005)  data  analyzing 
technical  teams.  The  other  three  stages  generally  occurred  at  a  specific  location  on  the 
timeline,  i.e.,  their  distribution  exhibited  a  well-formed  peak  on  the  timeline  much  like  that 
predicted  by  LaCoursiere  (1980).  Thus,  if  the  Storming  questions  were  changed  to  be  more 
sensitive  to  the  vigorous  (but  cooperative,  positive,  and  professional)  competition  of  ideas 
that  often  takes  place  within  a  technical  team,  there  may  be  more  of  this  newly  defined 
Storming  (e.g.,  cooperative  brainstorming)  but  perhaps  still  no  well-defined  Storming  stage. 

To  achieve  their  goals,  it  is  often  necessary  for  technical  team  members  to  challenge 
each  other.  Although  disagreements  and  divergent  points  of  view  were  common  among 
DAD  teams,  they  usually  were  resolved  quickly  within  a  cooperative  and  non-confrontational 
(minimal  friction,  resistance,  or  hostility)  atmosphere  according  to  their  technical  merits. 

This  type  of  professional  challenging  may  have  occurred  at  any  time  throughout  the  teaming 
process  but  did  not  cause  many  DAD  teams  to  exhibit  the  Storming  stage  as  defined  by  the 
Tuckman  model  and  as  represented  by  the  Miller  GPQ  (i.e.,  conflict,  resistance,  hostility  and 
friction).  The  two  Storming  questions  that  described  conflict  and  friction  (as  in  conflicting 
ideas,  and  the  friction  between  competing  viewpoints)  were  responsible  for  Storming 
behavior  being  lightly  (14%)  scattered  throughout  the  DAU  data.  The  Storming  questions 
that  focused  on  resistance  to  the  task  and  especially  the  one  focused  on  hostility  between 
team  members  were  not  relevant  to  the  observations  of  the  teams  being  studied. 

In  summary,  a  comparison  to  Benfield’s  (2005)  data  suggests  that  the  lack  of 
Storming  within  the  DAU  data  is  not  an  attribute  of  team  size,  duration,  or  team  setting. 

Thus,  it  is  suspected  that  the  lack  of  Storming  is  a  natural  attribute  of  technical  professionals 
working  under  time  constraints  to  produce  good-quality  products  for  which  they  are  held 
collectively  responsible.  The  technical  team  setting  of  this  research  and  Benfield’s  (2005) 
research  is  dramatically  different  in  form,  purpose  and  content  than  the  dominant  setting 
(therapy  groups)  used  by  Tuckman  (1965).  It  seems  reasonable  that  Storming,  as  Tuckman 
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(1965)  defined  it  and  Miller  (1997)  implemented,  would  occur  more  often  in  a  therapy  group 
setting  emphasizing  personal  interaction  than  in  a  technical  team  setting  emphasizing 
professional  interaction  where  each  team  member’s  personal  success  is  dependent  upon 
the  collective  success  of  the  team. 

Performing  Sequences  Analysis  for  the  F<N<P  three-stage  (  =0.05)  model  revealed 
that  229  (71%)  of  the  321  teams  generated  statistically  valid  sequences  that  followed  the 
F<N<P  three-stage  model.  Of  these,  161  (50  %)  teams  also  produced  an  F<N<P  average 
time-of-occurrence  sequence  of  stages.  Also,  637  (44%)  of  the  1 ,448  individuals 
experienced  a  statistically  valid  F<N<P  sequence.  This  variant  does  clearly  constitute  a 
majority  model  of  team  behavior.  Because  almost  3/4  of  the  DAD  teams  experienced  a 
statistically  valid  F<N<P  sequence,  the  F<N<P  model  is  a  reasonably  strong  contender  for  a 
general  model  of  technical  acquisition  team  dynamics.  I  refer  to  the  F<N<P  model  as  the 
DAD  model. 

Certainly,  more  research  is  required  to  evaluate  the  causal  connection  between  a 
team’s  productivity  and  its  experience  of  the  F<N<P  development  process.  More  work  will 
be  needed  to  assess  the  efficacy  and  general  applicability  of  guiding  a  team  through  the 
F<N<P  development  process  In  order  to  enhance  its  performance.  If  the  definition  and 
description  of  Storming  is  generalized  in  the  survey  instrument  to  include  brain  storming, 
perhaps  it  too  would  play  a  part  in  developing  a  strategy  to  optimize  team  performance. 

Because  the  Norming  and  Performing  behaviors  seemed  to  be  intermingled  on  the 
timeline  (on  the  average,  their  means  are  separated  by  about  2.5  timeline  units), 
differentiating  between  the  first  (F<N<P)  and  second  (F<P<N)  most  commonly  experienced 
sequence  is  problematical.  Consequently,  a  two-stage  model  F<N/P  (Forming  occurs 
before  Norming,  and  Forming  occurs  before  Performing)  that  combines  both  should 
represent  the  single  most  widely  experienced  sequence.  The  Sequence  Analysis  (  =0.05) 
was  applied  to  the  two-stage  model  F<N/P.  The  results  indicate  that  290  (90.34  %)  of  the 
321  teams  had  a  statistically  valid  experience  of  the  F<N/P  sequence.  This  variant  clearly 
constitutes  a  strong  model  of  DAD  team  behavior.  In  addition,  895  (62%)  of  the  1448 
individuals  also  experienced  a  valid  F<N/P  sequence.  Unfortunately,  a  simple  two-stage 
model  (first  a  team  experiences  Forming,  and  then  it  experiences  everything  else)  does  not 
provide  much  information  about  how  one  might  possibly  optimize  team  productivity  other 
than  make  sure  that  every  team  thoroughly  accomplishes  Forming  at  its  beginning. 

Figure  8  shows  that  for  all  three  sequence  models,  above-average  teams  produced 
the  most  statistically  significant  results  followed  by  average  teams,  while  below-average 
teams  produced  the  fewest  statistically  significant  results.  The  data  shows  consistent 
descending  stair-stepped  results  in  quantity  of  sequences  generated  for  each  team 
dynamics  model  as  the  teams’  rating  moves  from  above  average  to  below  average. 
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Figure  8.  Instructor  Evaluation  vs.  Teams  Producing  Statistically  Significant 

Sequences 

A  chi  square  r  x  c  contingency  test  was  performed  to  determine  the  correlation 
between  instructor  assessment  and  a  team’s  probability  of  producing  one  of  the  three 
sequences  of  Tuckman  stages  (F<S<N<P,  F<N<P  or  F<N/P).  The  correlation  numbers 
given  in  Figure  8  are  the  probabilities  that  the  populations  are  not  independent — i.e.,  the 
probability  that  there  is  a  relationship  between  a  team’s  performance  and  the  model  of  team 
dynamics  followed  by  that  team.  Correlations  of  0.95  or  greater  are  considered  to  represent 
a  relationship  between  populations  that  is  statistically  significant.  The  more  productive  and 
successful  a  team  was,  the  more  likely  they  were  to  observe  one  of  the  three  sequences  of 
Tuckman  stages  assessed  by  this  research. 

After  generating  a  distribution  of  stage  time-of-occurrence  data,  it  was  noticed  that 
the  stage  times-of-occurrence  for  all  321  teams  tended  to  group  together.  In  other  words,  all 
the  DAD  teams,  regardless  of  their  task  or  duration,  experienced  the  Forming,  Norming,  and 
Performing  stages  at  about  the  same  place  on  the  50-unit  timeline.  To  verify  this 
phenomenon,  the  Kruskal-Wallis  test,  as  described  by  Conover  (1999)  was  used  to 
determine  if  an  ensemble  of  the  DAD  time-of-occurrence  data  generated  by  each  of  the 
1448  individuals  for  each  Tuckman  question  could  be  separated  into  discrete  stages.  The 
data  indicate  that  an  ensemble  of  all  DAU  team  members  from  all  teams  do  collectively 
experience  a  discrete  sequence  of  at  least  three  Tuckman  stages.  This  result  corroborates 
the  possibility  of  a  universal  experience  of  the  Forming,  Norming,  and  Performing  stages  of 
the  Tuckman  model  (Tuckman  variant  1,  F<N<P,  DAU  Model)  at  a  somewhat  predictable 
fraction  of  a  team’s  duration.  However,  the  Storming  data  was  spread  across  the  entire 
timeline,  producing  no  distinct  peak.  Forming  appears  to  occur  at  about  25%  of  the  timeline, 
Norming  at  about  40%  of  the  timeline,  and  Performing  at  about  45%  of  the  timeline. 

Primary  Conclusions 

The  development  of  technical  acquisition  teams  appear  to  follow  a  variant  of  the 
Tuckman  model  (F<S<N<P).  This  model,  which  I  will  call  the  DAU  Team  Dynamics  model, 
has  three  discrete  stages  (F<N<P)  and  one  continuous  brainstorming  stage  that  takes  place 
over  the  entire  duration  of  the  team  effort.  The  brainstorming  activity  can  be  described  as 
group  members  challenging  each  others’  ideas  and  approaches  in  a  cooperative  way  with 
the  intention  of  producing  a  better  product  or  improving  the  team’s  process  (efficiency  and 
productivity). 

This  research  demonstrates  that  not  only  do  technical  teams  follow  the  DAU  model, 
but  that  teams  following  the  DAU  model  produce  better  products  than  teams  that  do  not 
follow  this  model.  It  may,  therefore,  be  possible  to  significantly  improve  productivity  in 
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technical  teams  by  facilitating  the  DAU  model — that  is,  to  encourage  teams  to  first  coalesce 
as  a  team  and  form  their  intent  and  structure,  then  develop  their  approach,  ground  rules, 
and  processes,  to  be  followed  by  assigning  tasks  and  getting  the  work  done — all  the  while 
cooperatively  challenging,  re-evaluating,  and  improving  the  overall  team  process  as  they 
work  together  to  accomplish  the  task  they  were  given.  Additionally,  one  should  expect  the 
Forming  stage  of  the  DAU  model  to  occur  at  about  25%  of  the  timeline,  the  Norming  stage 
to  occur  at  about  40%  of  the  timeline,  and  the  Performing  stage  to  occur  at  about  45%  of  the 
timeline.  Establishing  a  firm  causality  between  following  the  development  structure  of  the 
DAU  model  and  improving  a  technical  team’s  productivity  will  require  additional 
corroborating  research. 

Secondary  Conclusions 

The  tools  and  methods  developed  in  this  research  project  are  widely  applicable  to  a 
broad  assortment  of  team-dynamics  research  projects.  Furthermore,  developing  a  custom 
set  of  tools  to  fit  each  individual  research  application  is  not  difficult.  These  two  facts  should 
encourage  much  additional  research. 

Though  learning  how  to  make  teaming  more  efficient  and  productive  has  always 
been  considered  of  vital  importance  to  large  numbers  of  users,  the  research  process  has 
been  so  cumbersome,  difficult,  inconsistent,  and  lengthy,  that  the  field  has  languished 
(relative  to  its  importance)  for  decades.  Now  that  this  research  project  has  developed  a 
statistically  and  scientifically  rigorous  process  that  enables  the  assessment  of  a  large 
number  of  teams  relatively  easily  and  quickly,  it  is  hoped  that  the  pace  of  progress  will 
accelerate.  The  analysis  engine  and  methodology  developed  for  this  project  provides  a 
general  model  for  facilitating  low-budget,  quick  turn-around,  high-yield,  and  statistically 
rigorous  research  focusing  on  various  team  types,  settings,  sizes,  durations,  compositions, 
and  configurations.  Fortunately,  an  instrument  and  its  associated  analysis  engine  once 
developed  can  easily  be  used  by  others  to  perform  similar  research  in  different  settings,  with 
different  populations,  with  different  types  of  tasks,  and  with  teams  of  different  sizes  and 
durations. 
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Abstract 

Overview 

Teams  can  be  a  significant  resource  to  business  leaders  and  can  help  lead  to 
greater  program  successes.  Little  empirical  data  exist  on  what  strategic  characteristics 
make  teams  more  effective.  This  study  was  conducted  on  57  student  project  teams  in  12 
classes  (327  respondents)  in  a  Defense  Acquisition  University  (DAU)  executive  level,  six- 
week  program  management  class  in  six  different  locations.  The  study  not  only  underscores 
the  significance  of  team  focus  on  performance  but  also  highlights  how  team  characteristics 
affect  team  focus  and  performance.  The  results  of  this  study  have  applications  to  the 
successful  use  of  project  teams  throughout  the  DoD  and  in  the  commercial  industrial 
workplace. 


Results 


Significant  direct  relationships  were  found  in  the  15  tested  hypotheses  between  work 
team  strategic  intent  and  team  performance  as  measured  by  team  self-assessments  and 
instructor  assessments.  There  was  also  found  to  be  a  relationship  between  the  team  self- 
assessment  of  performance  and  the  instructors’  assessment  of  the  team  performance. 

This  study  provided  empirical  evidence  on  the  significant  relationships  between  work 
team  strategic  intent  and  work  team  performance.  The  research  accomplished  the 
following: 


ACQUISITIQN  RESEARCH:  CREATING  SYNERGY  EQR INEQRMED  CHANGE  -  91  - 


1 .  Provided  empirical  data  on  the  positive  correlation  relationships  between  work  team 
strategic  intent  and  work  team  performance. 

2.  Defined  characteristics  that  were  used  to  determine  the  strategic  intent  of  a  work 
team  or  any  work  unit. 

3.  Created  a  survey  to  measure  strategic  intent  of  team  members  and  teams  in  general. 

4.  Introduced  the  study  of  strategic  thinking  or  use  of  strategic  intent  as  a  method  for 
evaluating  team  performance. 

Introduction 

Organizations  operating  in  today’s  complex,  changing  and  sometimes  chaotic  work 
environments,  both  in  the  government  and  commercial  industries,  appear  to  be  more  and 
more  dependent  on  using  work  teams  to  leverage  themselves  to  be  more  creative,  efficient, 
and  focused.  Warren  Bennis  (1985),  in  his  book  Leaders,  The  Strategies  for  Taking 
Charge,  describes  the  need  for  cooperation,  communication,  and  collaboration  between 
individuals  in  order  to  achieve  greatness — and  emphasizes  the  successful  deployment  of 
teams  in  the  last  two  decades  to  achieve  these  same  results. 

In  today’s  society,  as  complex  and  technologically  sophisticated  as  it  is,  the  most 
pressing  projects  require  the  committed,  coordinated,  and  connected  contributions  of  many 
talented  people.  Gone  is  the  myth  of  the  Lone  Ranger  who  can  work  alone  and  is  larger- 
than-life.  Tomorrow’s  competitive  organizations  will  be  managed  and  inspired  by  teams  of 
experts,  skilled  technicians,  and  team-appointed  leaders.  Projects,  work  efforts,  and  entire 
programs  will  be  accomplished  by  a  network  of  linked,  disciplined  workers  skilled  in  their 
own  right  but  connected  by  their  commitment  to  their  team’s  greater  cause,  goals,  and/or 
objectives  (Bennis  &  Biederman,  1997). 

In  the  classic  written  about  teams.  The  Wisdom  of  Teams,  authors  Jon  R. 

Katzenbach  and  Nicholas  K.  Smith  identify  numerous  teams  in  various  industries  (Citibank, 
General  Electric,  Hewlett-Packard,  etc.)  that  have  been  continually  successful  in  employing 
high-performing,  self-managed  work  teams  (2003).  They  state  that  a  real  team  that  is 
appropriately  focused  and  rigorously  disciplined  is  the  most  versatile  unit  an  organization 
has  for  meeting  both  performance  and  change  challenges  in  today’s  complex  global 
markets. 

The  use  of  teams  has  been  increasing  for  the  last  20  years.  In  recent  data  collected 
from  Fortune  1000  companies,  it  was  highlighted  that  the  use  of  self-managed  teams  has 
increased  from  28%  in  1987  to  68%  in  2003  (Lawler,  Mohrman  &  Ledford,  1995).  A  study  of 
related  research  on  “self-managed  team”  underscored  the  fact  that  nearly  every  major 
American  corporation  is  considering  adopting  self-managed  team  as  an  organizational 
design  somewhere  in  their  organization  (Manz  &  Sims,  1993;  Wellins,  Byham  &  Wilson, 
1991). 

In  addition,  a  GAO  study  in  April  2001  highlighted  the  specific  advantages  of  using 
Integrated  Product  Teams  (IPT)  as  a  “best  practice”  to  improve  how  the  Department  of 
Defense  develops  and  acquires  weapon  systems  (2001,  April  10).  The  report  identifies 
specifically  the  successful  use  of  IPTs  by  the  military  in  the  Advanced  Amphibious  Assault 
Vehicle  Program  (AAAV)  to  reduce  the  time  needed  to  reduce  a  system  design  decision 
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from  6  months  to  about  a  week.  It  also  highlighted  three  commercial  companies  that 
effectively  used  high-performing,  self-managed  teams  to  improve  their  product  development 
capabilities.  Those  were:  Daimler  Chrysler,  Hewlett-Packard,  and  3M. 

Organizational  changes  are  occurring  with  ever-increasing  frequency,  and  the  scope 
of  change  is  often  revolutionary.  Effective  strategic  management  can  help  deal  with  this 
turbulence  and,  in  many  cases,  has  become  a  key  factor  for  organizational  success. 
Organizations  must  be  flexible  and  able  to  respond  to  these  environmental  challenges  with 
their  strategic  processes,  implementation  procedures,  and  organizational  structure. 

An  important  aspect  of  strategic  management  implementation  is  how  this  critical 
information  and  way  of  doing  business  is  being  infused  into  the  operational  end  of  the 
business.  Strategic  management  implementation  is  extremely  challenging;  lasting 
implementation  is  usually  the  exception,  not  the  rule,  due  to  the  resistance  to  this  change  of 
doing  business  and  the  many  layers  of  the  organization  that  must  be  touched  with  this 
enhanced  way  of  doing  business. 

An  effective  method  to  implement  strategic  management  is  through  work  teams  that 
are  focused  or  intent  with  the  same  strategic  goals  and  missions  of  the  corporate  leadership. 
Teams  with  a  significant  level  of  the  same  strategic  focus  on  the  purpose,  objectives,  and 
execution  strategies  that  are  aligned  with  the  corporate  goals  and  missions  can  be  an 
extremely  effective  implementation  tool  for  the  organization.  This  applies  in  a  similar  nature 
to  a  student  work  team  in  the  classroom,  attempting  to  learn  new  skills  and  knowledge. 

It  was  hypothesized  in  this  study  that  if  student  team  members  are  aligned  in  their 
purpose  and  objectives  to  the  course  goals  and  learning  objectives,  then  higher  levels  of 
student  team  performance  or  learning  will  result  that  is  aligned  more  directly  with  the 
course’s  and  the  instructors’  learning  objectives.  The  team’s  understanding  of  and 
commitment  to  the  course’s  purpose,  objectives,  and  strategies  may  help  ensure  that  the 
team  is  effectively  achieving  the  reason  for  being  in  the  course,  learning  and  performing  the 
course’s  goals  and  objectives.  An  objective  may  be  to  ensure  that  the  students  are  aligned 
to  the  same  strategic  focus  or  intent  as  the  instructors’  and  course  managers’  goals  of 
achieving  the  overall  course  objectives. 

Business  and  education  leaders  have  been  faced  with  an  increasingly  changing 
environment,  which  increases  the  need  for  effective  and  focused  strategic  planning  and 
implementation  practices.  The  speed  and  volume  of  transformation  have  increased 
dramatically  in  the  last  century,  and  this  trend  is  expected  to  continue  at  an  even  greater 
speed  and  impact  in  the  21  century.  Speed  and  change  are  expected  to  continue,  and  with 
this  challenge,  more  innovative  and  effective  strategic/future-oriented  measures  must  be 
achieved  by  business  and  education  leaders  to  ensure  they  are  able  to  sustain  their 
operations  and  maintain  the  proper  strategic  focus  and  intentions.  The  increased  level  of 
change  and  the  need  to  apply  effective  structural  change  initiatives,  such  as  teams,  to  this 
environment  of  change  prompted  the  interest  in  this  study. 

This  research  study  obtained  empirical  data  from  classroom  surveys  administered  to 
student  work  team  members  attending  a  6-week  Program  Managers’  PMT  352B  course  at 
Defense  Acquisition  University  (DAU)  and  to  DAU  PMT  352B  faculty  members  during  a 
year’s  period  of  time  (July  2005  to  July  2006).  The  data  from  the  surveys  determined  each 
respective  student  team’s  Strategic  Intent  and  each  team’s  Self-assessed  Team 
Performance  and  Instructor-assessed  Team  Performance.  Pearson’s  r  correlation 
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coefficient  and  Spearman’s  rho  ranked-order  correiation  tests  were  used  to  determine  the 
relationship  between  team  Strategic  Intent  variables  and  Team  Performance — both  team 
self-assessed  and  instructor-assessed.  This  paper  presents  a  summary  of  the  reasons  for 
the  study  and  the  results  of  the  study  which  estabiished  empirical  data  to  support  the 
general  hypothesis  that  increases  in  work  team  strategic  intent  or  focus  will  cause  increases 
in  the  work  team’s  performance. 

Past  research  has  provided  some  generai  discussions  and  initiai  studies  on  the 
relationship  between  team  characteristics  such  as  Strategic  Intent  and  Team  Performance. 
Previous  research  has  aiso  identified  the  need  for  further  research,  and  empiricai  data  were 
needed  in  this  area  of  research  to  determine  and  measure  the  reiationships  between  the 
variables  of  team  strategic  thinking  (Strategic  Intent)  and  Team  Performance  (Athanassiou, 
Crittenden  &  Kelly,  2000;  Bartlett  &  Ghoshal,  1998;  Hamel  &  Prahaland,  1994;  Thompson  & 
Strickland,  1996;  Tregoe  &  Zimmerman,  1989). 

Katzenbach  and  Smith  (2003)  have  accomplished  extensive  work  in  the  study  of 
teams  and  their  effectiveness.  They  admitted  that  no  empiricai  data  exist  to  prove  their 
theories  on  team  effectiveness.  This  research  study  provides  data  to  support  Katzenbach 
and  Smith’s  study  (2003)  and  theories  on  teams:  teams  can  more  effective  or  perform 
better  if  they  maintain  a  Strategic  Intent  or  focus  that  was  understood  and  committed  to  by 
ail  the  team  members.  This  paper  highiights  the  purpose  of  the  study,  the  key  concepts 
studied,  the  research  questions  and  hypotheses,  the  resuits  and,  finally,  some  conclusions 
reached  from  the  study. 

Purpose  of  Study 

The  purpose  of  this  study  was  to  use  survey  data  from  student  work  teams  and 
instructors’  surveys  to  examine  the  relationship  between  work  team  Strategic  Intent 
(strategic  purpose,  objectives,  and  strategies)  and  work  team  Performance.  The  studied 
work  teams  were  chosen  from  student  work  teams  attending  Defense  Acquisition  University 
(DAU)  PMT  352B  course.  PMT  352B  is  a  6-week-long  course  which  teaches  the  concepts 
and  skills  of  being  successful  program  managers.  It  simulates  the  conditions  and  stresses 
that  senior  DoD  managers  are  normally  presented  with  in  making  daily  and  long-term 
strategic  program  management  decisions.  Team  Performance  was  assessed  by  surveys 
administered  to  the  work  teams  (self-assessment  performance)  and  to  the  PMT  352B 
instructors  who  were  teaching  the  student  work  teams  (external,  instructor  assessment). 

The  specific  strategic  elements  studied  included  the  teams’  strategic  purpose, 
objectives,  and  strategies  that  had  been  determined  in  previous  academic  and  business 
research  (Ackoff,  1974;  Ansoff  &  McDonnell,  1990;  Anthanassiou  etal.,  2000a,  2000b; 

Elrod,  1999;  Hackman  &  Wageman,  2005;  Kraft,  1996;  Schein,  1980;  Thompson,  1993), 
related  to  the  decision-making  success  of  an  organization  and  having  an  effect  on 
organizational  performance  and  long-term  successes. 

Strategic  thinking  and  alignment  of  this  thinking  have  been  used  in  past  research  to 
measure  a  team’s  ability  to  agree  among  the  members  on  strategic  goals,  objectives,  and 
strategies  that  focus  on  or  align  the  team’s  efforts  on  shared  performance  objectives 
(Anthanassiou  et  al.,  2000a).  This  alignment  or  cohesiveness  in  strategic  thinking  in  the 
Anthanassiou  et  al.  studies  (2000a,  2000b)  was  measured  to  determine  the  difference  or 
variance  between  the  leader’s  and  the  team’s  perceptions  and  commitments  to  the  same 
strategic  elements  (team  goals,  objectives,  and  strategy).  They  were  studied  to  determine 
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the  effects  on  performance.  In  most  cases,  the  higher  the  alignment  or  congruence  in 
strategic  thinking,  the  better  the  performance  of  the  team  and  its  associated  business 
outcomes  (financial  and  social). 

This  research  study  acquired  empirical  data  from  student  work  team  members 
attending  classes  in  PMT  352B.  The  strategic  characteristics  of  specific  PMT  352B  student 
work  teams  were  calculated  from  information  gathered  from  team  surveys.  These  students 
were  mature  (generally  35  to  60  years  of  age)  Department  of  Defense  (DoD)  students 
attending  this  technical  training  course  on  program  management  with  Defense  Acquisition 
University.  The  teams’  understanding  of  and  commitment  to  their  respective  team’s 
strategic  management  characteristics  was  measured  by  surveys  administered  to  the  teams 
in  their  location  of  work  (the  classroom)  by  the  researcher  and  trained  faculty  members.  The 
surveys  obtained  each  team  member’s  perceptions  of  his  understanding  of  and  commitment 
to  the  specific  team  strategic  elements  studied  in  this  research — team  purpose,  objectives, 
and  strategies.  These  strategic  elements  helped  define  the  teams’  strategic  characteristics 
and  were  defined  in  the  team  survey,  so  there  was  an  understanding  of  these  variables  by 
the  survey  respondents.  This  helped  define  what  were  the  strategic  elements  being  studied 
and  what  were  the  data  the  researcher  was  seeking. 

Data  were  collected  from  each  team  member  on  his  perception  of  how  similar  or 
linked  was  his  understanding  of  and  commitment  to  the  team  compared  to  the  other 
members  of  the  team’s  understanding  of  and  commitment  to  the  team’s  purpose,  objectives, 
and  strategies.  Team  similarity  was  measured  both  in  terms  of  understanding  and 
commitment  to  these  strategic  elements. 

The  research  calculated  team  data  on  team  similarity  of  team  strategic 
characteristics  as  measured  by  understanding  and  commitment  to  team  purpose,  objectives, 
and  strategies.  The  research  analyzed  the  relationship  of  these  strategic  characteristics 
(similar  understanding  of  and  commitment  to  team  purpose,  objectives,  and  strategies)  to 
Team  Performance — measured  by  the  team’s  self-assessment  of  its  performance  and  by  an 
external  assessment  by  the  team’s  instructor(s).  The  study  then  analyzed  the  relationship  or 
similarity  between  a  team’s  self-assessment  of  its  performance  and  the  instructors’  external 
assessment  of  the  same  team’s  performance.  The  researcher  theorized  that  the  similarity  or 
alignment  of  a  team’s  purpose,  objectives,  and  strategies  was  a  strong  predictor  (a  direct 
correlation)  of  how  well  the  team  members  worked  together  and  effectively  communicated  in 
making  critical  choices  vital  to  the  successful  performance  of  the  team.  Team  effectiveness 
in  making  decisions  and  accomplishing  the  course  objectives  was  theorized  to  be  related  to 
the  congruence  or  alignment  of  each  team  member’s  individual  similarity  perceptions  of  his 
strategic  characteristics  to  the  other  members  on  the  team. 

This  congruence  was  measured  in  terms  of  the  member’s  understanding  of  and 
commitment  to  the  other  team  members’  strategic  elements  of  purpose,  objectives,  and 
strategies.  How  congruent  or  similar  the  members’  strategic  characteristics  were,  the  more 
effective  the  team  should  be  in  accomplishing  its  purpose,  objectives,  and  strategies. 
Accomplishing  these  team  strategic  elements  would  make  the  team  perform  better,  both  as 
determined  by  the  team’s  own  standards  and  by  the  instructors’  criteria  of  learning  the 
course  objectives.  The  following  research  model  in  Figure  1  helped  to  identify  the  variables 
(independent  and  dependent),  research  questions,  hypotheses,  and  relationships  involved 
in  this  research  study.  The  next  two  sections  highlight  the  two  key  variables  studied: 
Strategic  Intent  and  Team  Performance. 
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strategic  Intent 

The  research  model  in  Figure  1  highlights  the  key  variables  and  relationships  studied 
in  this  research.  The  Strategic  Intent  of  the  team  is  defined  and  highlighted  in  the  figure  as 
consisting  of  three  team  strategic  elements:  purpose,  objectives,  and  strategies.  Strategic 
Intent  is  further  defined  as  to  how  each  team  member  was  focused  or  had  similarly  aligned 
understanding  of  and  commitment  to  the  team’s  strategic  elements  (purpose,  objectives, 
and  strategies),  as  measured  by  surveying  each  team  member.  The  actual  measurement  of 
Strategic  Intent  was  then  computed  by  measuring  the  overall  average  team  scores  for 
Strategic  Intent  from  the  individual  members’  scores  on  the  team  survey. 

One  of  the  basic  reasons  for  using  the  term  “Strategic  Intent”  to  highlight  the 
strategic  thinking  or  focus  of  the  teams  in  this  study  was  to  use  the  previous  work  of  Hamel 
and  Prahaland  (1989)  in  this  conceptual  or  research  area.  Strategic  Intent  captures  the 
meaning  and  nature  of  the  characteristics  most  representative  of  what  exists  in  teams  or 
other  groups  that  highlight  what  they  think  and  perceive  about  their  future  goals,  vision,  or 
purpose.  As  discussed  by  Hamel  and  Prahaland,  an  organization’s  Strategic  Intent  or  focus 
is  part  of  the  “dream  that  energizes  a  company  and  is  more  sophisticated  and  more  positive 
than  a  simple  war  cry”  (p.  64).  These  two  authors  highlighted  that  Strategic  Intent  implies  a 
sense  of  organizational  direction,  discovery,  and  destiny.  They  explained  that  Strategic 
Intent  is  more  than  the  implied  particular  point  of  view  about  the  long-term  market  or 
competitive  position  that  an  organization  hopes  to  build  over  the  coming  decade  or  so.  It  is 
the  stated  and  vital  focus  that  makes  an  organization  competitive  and  driven  toward  a  vision, 
a  future  direction,  or  a  destiny  that  consumes  its  nature  and  reason  for  being  (Hamel  & 
Prahaland,  1989).  These  are  the  characteristics  most  representative  of  what  this  research 
desired  to  study  and  why  the  research  was  originally  conducted. 

This  research  study  embraced  a  similar  meaning  and  value  to  team  Strategic  Intent 
developed  by  Hamel  and  Prahaland — the  committed  and  understood  strategic  elements  of 
the  team  that  united  or  focused  team  actions  and  decisions  as  measured  by  the  team’s 
commitment  to  and  understanding  of  the  team’s  purpose,  objectives,  and  strategies. 

In  this  research,  teams  were  considered  important  to  facilitating  strategy 
implementation  and  integration  when  properly  focused  on  the  organization’s  strategic 
purpose,  objectives,  and  strategies.  The  strategic  focus  or  intent  of  teams  was  studied  to 
determine  what  relationship  strategic  intent  has  on  overall  Team  Performance.  Studying 
this  relationship  in  teams  could  have  a  direct  bearing  on  how  these  same  variables 
(Strategic  Intent  and  Performance)  are  related  in  larger  organizations,  such  as  divisions, 
business  units,  plants,  and  firms. 

Adequate  controls  of  the  decision-making  processes  are  in  place  within  the  focused 
team,  which  facilitate  it  to  be  more  effective  and  successful  as  a  decision-maker  in  focusing 
on  the  team  purpose  and  objectives.  Additionally,  it  can  also  make  the  team  more 
integrated  and  focused  within  the  overall  organizational  structure,  enabling  or  leveraging  the 
organization  itself  to  be  higher  performing  in  the  long  term.  Properly  disciplined,  focused, 
and  integrated  teams  are  the  ones  that  become  high-performing  teams,  and  they  have  been 
considered  “the  most  versatile  unit  organizations  have  for  meeting  both  performance  and 
challenges  in  today’s  complex  world”  (Katzenbach  &  Smith,  2003,  p.  xiii). 

In  their  book.  Built  to  Last  Successful  Habits  of  Visionary  Companies,  Jim  Collins 
and  Jerry  Porras  (1994)  described  the  strategic  elements  or  Intent  needed  to  ensure  that 
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effective  strategic  decisions  can  be  made.  In  their  study  of  18  highly  successful  visionary 
companies,  Collins  and  Porras  highlighted  that  core  ideologies  are  relevant  to  making 
effective  strategic  choices.  It’s  what  “drives”  these  companies  to  conform  to  be  successful  in 
developing  new  products  and  services.  This  same  emphasis  on  Strategic  Intent  was 
previously  highlighted  by  Ansoff  and  McDonnell  (1990)  and  Mintzberg  (1994). 

The  premise  of  Collins  and  Porras’  1994  book  is  that  core  ideology  or  Strategic  Intent 
provides  the  foundation  for  the  continual  successes  of  the  18  visionary  companies,  the 
performance  of  which  the  researchers  tracked  over  a  6-year  period.  These  companies  were 
standouts  in  their  industry,  and  Collins  and  Porras  theorized  that  the  reasons  for  their 
continual  successes  were  directly  related  to  the  existence  of  a  core  ideology  upon  which  the 
firms  and  their  upper-echelon  management  teams  based  their  existence  and  strategic 
behavior.  Their  book  concludes  that  if  the  core  ideology  of  a  firm  and  its  strategic  thinking 
are  properly  aligned  with  the  environment,  the  firm  and  its  thinking  will  have  a  greater 
opportunity  to  be  successful  in  the  long  term,  and  it  will  bring  in  above-average  performance 
returns  and  profits. 

The  work  conducted  by  Collins  and  Porras  is  noteworthy  and  highlighted  the 
usefulness  of  Strategic  Intent  or  core  ideologies  in  determining  successful  performance. 
Their  premise  is  that  based  on  their  study  and  thinking  of  core  ideologies,  firms  are  able  to 
sustain  their  outstanding  performance  in  the  competitive  market  by  staying  focused  on  their 
core  ideologies.  Their  study,  although  popular  with  business  leaders,  is  limited  in  the 
empirical  sense  since  no  hard  data  exist  in  their  studies  that  prove  or  empirically  support 
their  theories.  They  have  significant  anecdotal  information  and  cases  but  not  empirical  data. 
As  discussed,  the  purpose  of  this  study  was  to  help  identify  and  collect  empirical  data  on  the 
effects  of  Strategic  Intent  on  performance. 

Team  Performance 

The  concept  of  Team  Performance  and  how  to  measure  it  is  critically  important  to  the 
successful  deployment  of  teams  in  any  environment  (Kraft,  1996).  There  is  the  general 
belief  that  teams  make  organizations  more  effective.  However,  few  research  efforts  have 
measured  team  effectiveness  with  empirical  data.  The  research  cited  in  this  study  focused 
primarily  on  the  manufacturing  teams  that  can  be  assessed  using  operational  measures 
such  as  productivity,  efficiency,  delivery  time,  defects,  and  scrap  (Beyerlein,  1995).  Some  of 
the  challenges  presented  in  this  research  study  on  measuring  Team  Performance  were 
similar  to  many  studies  that  relied  upon  self-reported  assessments,  especially  when 
measuring  Team  Performance.  Team  Performance  has  been  studied  extensively,  and 
many  techniques  exist  to  measure  it.  How  to  measure  Team  Performance  in  the  classroom 
or  even  in  a  program  office  environment  is  a  challenge  without  using  self-reported  or  self- 
assessed  performance  measures  or  data. 

Podsakoff  and  Organ’s  (1986)  work  highlighted  that  the  most  critical  concern  was 
that  the  use  of  self-reports  was  identifying  the  potential  causes  of  “artifactual  covariance” 
between  self-report  measures  of  what  were  presumed  to  be  two  distinctly  different  variables. 
They  stressed  that  when  the  same  persons  provided  the  researcher  with  self-report 
measures  of  two  or  more  different  constructs,  what  could  account  for  any  correlations  that 
were  found  in  the  research  could  be  a  result  of  the  “artifactual  covariance”  and  bias  in  the 
respondents’  self-reported  data  and  not  the  natural  correlation  between  the  variables.  In 
other  words,  false  correlations  would  be  assumed  in  the  research,  based  on  the  nature 
(“artifactual  covariance”)  of  the  self-reported  data  and  not  on  the  actual  relationship 
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occurring  in  the  cases  analyzed.  The  authors  highiighted  several  techniques  to  reduce  the 
effects  of  this  overiapping  influence  of  the  respondents’  self-reported  data.  All  of  these 
methods  were  highlighted  as  a  means  to  reduce  the  effects  of  obtaining  data  from  seif- 
reports. 

The  Podsakoff  and  Organ  (1986)  article  highlighted,  though,  that  under  specific 
conditions  it  seems  that  seif-report  data  in  organizational  studies  are  “here  to  stay.”  They 
aiso  reported  on  another  study  by  Howard,  Maxwell,  Weiner,  Boynton  and  Rooney  .  (1980), 
which  noted  that  under  many  circumstances,  self-reports  might  represent  more  accurate 
estimates  of  population  parameters  than  behavioral  measures.  Podsakoff  and  Organ  also 
stressed  in  their  study  that  it  is  uniikely  that  such  techniques  of  using  seif-reports  wiii  be 
abandoned.  They  do  recommend  that  caution  be  taken  to  ensure  that  the  right  conditions 
exist  to  minimize  the  effects  of  self-reported  data  on  the  correlations  and  conclusions  made 
concerning  the  data  reiationships.  Gupta  and  Behr  (1982)  emphasized  that  despite  the 
problems  in  the  use  of  self-report  measures  in  organizational  research,  the  practical  utility  of 
seif-reported  data  makes  them  a  necessity  to  organizationai  behavior  studies.  Seif-report 
data  are  extremeiy  usefui  and  make  them  “virtuaily  indispensabie  in  many  research 
contexts”  (Podsakoff  &  Organ,  1986,  p.  540). 

In  the  context  of  this  research,  the  application  and  usefuiness  of  self-reported  data 
from  team  members  attending  PMT  352B  courses  are  justified,  based  on  the  fact  that  the 
seif-reported  data  are  coilected  at  different  iocations/settings,  at  different  times,  and  using 
consistent  but  varying  instructors  in  gathering  the  data  and  that  the  data  are  aggregated  at 
the  team  ievei  (one  level  above  the  team-member  level,  where  the  data  were  originally 
gathered).  These  are  all  methods,  as  explained  by  Podsakoff  and  Organ  (1986),  which 
reduce  the  negative  effects  of  self-reported  data. 

The  nature  of  the  data  used  in  this  research  also  necessitated  that  to  obtain  team 
characteristics  on  Strategic  Intent,  the  natural  source  of  the  information  wouid  be  from  the 
team  members.  The  team  members  were  the  most  reiiable  source  of  information  on  how 
they  thought  about  the  Team  Performance  and  how  simiiar  they  perceived  their  beliefs  to  be 
regarding  team  purpose,  objectives,  and  strategies  (Strategic  Intent).  It  would  be  difficult,  if 
not  impossibie,  to  obtain  “true”  unbiased,  objective  data  on  teams’  perceptions  of  their 
strategic  thinking  and  their  performance  without  using  seif-reported  data. 

The  effects  of  self-reported  data  have  been  assessed  in  this  research.  It  was 
determined  that  given  the  nature  of  the  seif-reported  team  member  data  (aggregated  at 
team  level,  collected  from  different  sources,  locations,  and  times),  the  effects  of  “artifactual 
covariance,”  as  highiighted  by  Podsakoff  and  Organ  (1986),  were  minimized  in  this 
research. 

It  is  evident  that  the  problems  of  measuring  Team  Performance  are  very  complex 
and  difficult  to  pinpoint.  The  existing  performance  measurement  systems  in  place  in  an 
organization  are  usualiy  not  aligned  with  new  initiatives  or  changes,  such  as  team 
deveiopment,  occurring  in  today’s  workplace.  In  most  of  these  cases,  the  measurement 
systems  do  not  adequately  reflect  the  impact  on  efficiency  and  effectiveness  of  the  latest 
initiatives  (Beyeriein,  1995).  Because  of  these  many  difficulties  with  the  lack  of  integrated 
performance-measurement  systems  and  the  complexities  of  how  teams  affect  organizations, 
it  is  difficuit,  if  not  impossible,  to  effectively  measure  the  value  of  teams  with  existing 
databases  or  performance-management  systems.  Therefore,  it  is  beiieved  that  the  oniy 
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effective  way  to  measure  Team  Performance  is  through  seif-assessment  of  Team 
Performance.  Lets  know  understand  the  study’s  research  questions  and  hypotheses. 


Research  Questions  and  Hypotheses 

This  study  was  concerned  with  a  general  broad  research  question  that  focused  on 
determining  the  relationship  between  work  team  Strategic  Intent  and  Team  Performance. 
The  purpose  of  the  study  was  to  complete  a  thorough,  descriptive,  correlational  relationship 
study  on  the  six  team  Strategic  Intent  variables  dealing  with  the  teams’  strategic  thinking 
and  on  the  performance  of  the  teams  as  assessed  by  the  team  members  themselves  and  by 
the  teams’  instructors  in  PMT  352B  program  management  courses. 

The  research  study  gathered  empirical  evidence  to  answer  these  research  questions 
and  provided  data  to  support  these  hypotheses: 

RQ1.  What  was  the  relationship  between  the  overall  Team’s  Strategic  Intent  and  the 
overall  Team’s  Performance  (team  member  self-assessment)? 

Ha1.  There  was  a  statistically  significant  relationship  between  the  overall  Team’s 
Strategic  Intent  (Independent  variable/interval  data)  and  the  overall  Team’s 
Performance  (team  member  self-assessment)  (Dependent  variable/interval 
data).  The  statistical  test  used  was  the  Pearson’s  r  correlation  coefficient. 

RQ2.  What  was  the  relationship  between  the  Team’s  Strategic  Intent  as  measured 
by  the  two  variables  a-b  below  and  Team  Performance  (team  member  self- 
assessment)? 

a.  Understanding  of  Team  Purpose 

b.  Commitment  to  Team  Purpose 

Ha2.  There  was  a  statistically  significant  relationship  between  the  Team’s  Strategic 
Intent  as  measured  by  the  two  variables  (Independent  variable/interval  data) 
a-b  below  and  Team  Performance  (team  member  self-assessment) 
(Dependent  variable/interval  data).  The  statistical  test  used  was  the 
Pearson’s  r  correlation  coefficient. 

a.  Understanding  of  Team  Purpose 

b.  Commitment  to  Team  Purpose 

RQ3.  What  was  the  relationship  between  the  Team’s  Strategic  Intent  as  measured 
by  the  two  variables  a-b  below  and  Team  Performance  (team  member  self- 
assessment)? 

a.  Understanding  of  Team  Objectives 

b.  Commitment  to  Team  Objectives 

Ha3.  There  was  a  statistically  significant  relationship  between  the  Team’s  Strategic 
Intent  as  measured  by  the  two  variables  (Independent  variable/interval  data) 
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a-b  below  and  Team  Performance  (team  member  self-assessment) 
(Dependent  variable/  interval  data).  The  statistical  test  used  was  the 
Pearson’s  r  correlation  coefficient. 

a.  Understanding  of  Team  Objectives 

b.  Commitment  to  Team  Objectives 

RQ4.  What  was  the  relationship  between  the  Team’s  Strategic  Intent  as  measured 
by  the  two  variables  a-b  below  and  Team  Performance  (team  member  self- 
assessment)? 

a.  Understanding  of  Team  Strategies 

b.  Commitment  to  Team  Strategies 

Ha4.  There  was  a  statistically  significant  relationship  between  the  Team’s  Strategic 
Intent  as  measured  by  the  two  variables  (Independent  variable/  interval  data) 
a-b  below  and  Team  Performance  (team  member  self-assessment) 
(Dependent  variable/interval  data).  The  statistical  test  used  was  the 
Pearson’s  r. 

a.  Understanding  of  Team  Strategies 

b.  Commitment  to  Team  Strategies 

RQ5.  What  was  the  relationship  between  the  overall  Team’s  Strategic  Intent  and  the 
overall  Team’s  Performance  (external  instructor  assessment)? 

Ha5.  There  was  a  statistically  significant  relationship  between  the  overall  Team’s 
Strategic  Intent  (Independent  variable/interval  data)  and  the  overall  Team’s 
Performance  (external  instructor  assessment)  (Dependent  variable/interval 
data).  The  statistical  test  used  was  the  Pearson’s  r. 

RQ6.  What  was  the  relationship  between  the  Team’s  Strategic  Intent  as  measured 
by  the  six  variables  a-f  below  and  Team  Performance  (external  instructor 
assessment)? 

a.  Understanding  of  Team  Purpose 

b.  Commitment  to  Team  Purpose 

c.  Understanding  of  Team  Objectives 

d.  Commitment  to  Team  Objectives 

e.  Understanding  of  Team  Strategies 

f.  Commitment  to  Team  Strategies 

Had.  There  was  a  statistically  significant  relationship  between  the  Team’s  Strategic 
Intent  as  measured  by  the  six  variables  (Independent  variable/interval  data)  a- 
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f  below  and  Team  Performance  (external  instructor  assessment)  (Dependent 
variable/ interval  data).  The  statistical  test  used  was  the  Pearson’s  r 
Correlation  Coefficient. 

a.  Understanding  of  Team  Purpose 

b.  Commitment  to  Team  Purpose 

c.  Understanding  of  Team  Objectives 

d.  Commitment  to  Team  Objectives 

e.  Understanding  of  Team  Strategies 

f.  Commitment  to  Team  Strategies 

RQ7.  What  was  the  relationship  between  the  overall  Team’s  Performance  (self- 
assessment  from  team  survey)  and  the  overall  Team’s  Performance 
(Instructor  assessment  from  instructor  survey)? 

Ha7.  There  was  a  statistically  significant  relationship  between  the  overall  Team’s 
Performance  (self-assessment  from  team  survey)  and  the  overall  Team’s 
Performance  (instructor  assessment  from  instructor  survey).  The  statistical 
test  used  was  the  Pearson’s  r  correlation  coefficient. 

The  following  figure  highlights  the  relationships,  research  questions  and  hypotheses 
of  the  study. 
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Figure  1.  Research  Model  with  Hypotheses 


DETAILED  RESEARCH  MODEL:  WORK  TEAM  STRATEGIC  INTENT 
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Instrumentation 

There  were  two  surveys  involved  in  the  data  accumulation.  The  first  survey  was  the 
team  survey.  The  second  survey  was  the  instructor  survey.  A  team-member  survey 
questionnaire  consisted  of  scaled  questions.  The  scaled  questions  included  4-point  Likert 
scales  and  continuous  rating  scales  (1,  low/not  similar,  to  4,  high/extremely  similar)  on  the 
items  related  to  evaluating  the  similarity  of  the  strategic  elements  of  purpose,  objectives, 
and  strategies.  Five-point  Likert-scale  questions  were  used  to  evaluate  Team  Performance, 
both  on  the  team-member  survey  and  the  separate  instructor  survey  (1 ,  low/poor,  to  5, 
high/excellent).  The  team-member  questionnaire  was  divided  into  categories  related  to  the 
variables  in  the  study:  purpose,  objectives,  and  strategies.  Headings  and  numbering  of 
questions  were  used  to  segregate  the  categories.  In  both  surveys,  definitions  were  provided 
for  the  key  variables  to  aid  in  the  understanding  of  the  key  concepts  and  variables  in  this 
research  and  to  aid  in  the  accuracy  of  the  responses.  Demographic  information  on  the 
survey  participants  was  collected  on  the  team-members’  survey. 

The  Team  Survey  instrument  (see  Appendix  A)  was  divided  into  four  parts  and 
contained  14  questions.  Part  1  and  4  were  questions  that  collected  team  and  student 
information  respectively  and  required  short,  circled  answers.  Part  2  was  composed  of  six 
questions  related  to  Strategic  Intent  on  4-point  Likert  scales.  Part  3  was  composed  of  three 
questions  related  to  Team  Performance  on  5-point  Likert  scales. 

The  Team  Performance  was  also  measured  by  the  assigned  DAU  PMT  352B 
instructors.  The  Instructor  Survey  provided  data  for  measuring  the  Instructor-assessed 
Team  Performance.  The  first  four  questions  on  the  Instructor  Survey  asked  the  instructors 
for  data  on  the  team’s  name,  location  of  the  class,  and  dates  of  the  class.  Question  4  asked 
the  instructor  to  evaluate  or  assess  the  individual  team’s  performance  in  accomplishing  the 
course  objectives.  The  instructors  rated  the  teams  on  a  5-point  Likert  scale  of  poor  (1)  to 
excellent  (5).  Comments  were  also  requested.  The  next  section  discusses  the  sample  that 
was  studied. 

Research  Sample  Population 

Fifty-seven  data  points  or  teams  were  collected  from  the  population  of  teams 
enrolled  in  PMT  352B  courses.  The  acquired  empirical  data  from  12  PMT  352B  classes 
were  obtained  from  surveys  conducted  in  the  classroom  from  57  student  teams,  their 
respective  student  members  (327),  and  32  team  instructors.  The  students  were  Department 
of  Defense  (DoD)  career  acquisition  professionals  attending  the  technical  training  course 
(PMT  352B)  at  one  of  the  five  Defense  Acquisition  University  campuses.  This  research  was 
a  co-sponsored  DAU/Alliant  International  University  research  project.  The  instructors  were 
certified  DAU  instructors,  aged  35-60  years  of  age. 

Demographic  information  was  analyzed  from  the  surveys  also.  Pearson’s  r 
correlation  coefficient  tests  were  used  to  test  the  correlation  between  team  average  age  and 
team  average  years  of  experience  to  Strategic  Intent,  Team-assessed  Team  Performance, 
and  Instructor-assessed  Performance.  Spearman’s  rho  rank-order  and  Pearson’s  r 
correlation  coefficient  tests  were  used  to  determine  the  relationships  between  team 
educational  levels  to  Strategic  Intent,  Team-assessed  Team  Performance,  and  Instructor- 
assessed  Team  Performance  since  the  data  (Educational  Levels)  were  nominal  data. 
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Overall  Research  Summary  and  Findings 

Table  20  highlights  the  relative  strength  of  each  of  the  Pearson’s  r  tests  that  were 
conducted  in  the  study,  including  those  identified  in  additional  findings.  Appendix  B  contains 
the  actual  SPSS  test  results.  The  italicized  entries  below  identify  the  original  15  Research 
Question  hypotheses,  which  were  all  supported  at  the  .05  significance  level  with  their  p- 
values.  All  but  the  last  entry  (Question  9  to  Instructor  Performance)  were  supported  at  a  .01 
significance  level.  All  the  tests  were  supported  at  the  .05  significance  level. 

For  this  paper,  the  following  strength  of  the  relationship  or  support  was  used: 
Pearson’s  r  greater  than  .7  is  considered  a  strong  relationship;  from  .5  to  .699  it  is 
considered  a  moderate  relationship:  and  from  .3  to  .499  it  is  considered  a  modest 
relationship/support. 


Table  1.  Relative  Strength  of  Tested  Variables  in  This  Study 


Variables 

Rank 

Pearson’s  r 

p-value 

Results 

Question  8  (CQ)  TQ 

Qverall  Strategic  Intent 

1 

.921 

.000** 

Strongly  Supported 

Question  10  (CS)  TQ 

Qverall  Strategic  Intent 

2 

.884 

.000** 

Strongly  Supported 

Question  7  (UQ)  TQ 

Qverall  Strategic  Intent 

3 

.880 

.000** 

Strongly  Supported 

Question  6  (CP)  TQ 

Qverall  Strategic  Intent 

4 

.871 

.000** 

Strongly  Supported 

Question  9  (US)  TQ 

Qverall  Strategic  Intent 

5 

.817 

.000** 

Strongly  Supported 

Question  5  (UP)  TQ 

Qverall  Strategic  Intent 

6 

.793 

.000** 

Strongly  Supported 

Overall  Strategic  Intent  TO 
Overall  Team 

Performance 

7 

.731 

.000** 

Strongly  Supported 

Hypothesis  1 

Qverall  Strategic  Intent  TQ 
Question  1 1 

8 

.724 

.000** 

Strongly  Supported 

Qverall  Strategic  Intent  TQ 
Question  12 

9 

.706 

.000** 

Strongly  Supported 

Qverall  Strategic  Intent  TQ 
Question  13 

10 

.680 

.000** 

Moderately 

Supported 
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Variables 

Rank 

Pearson’s  r 

p-value 

Results 

Question  10  (CS)  TO 

Overall  Team 

Performance 

11 

.673 

.000** 

Moderately 

Supported 

Question  9  (US)  TO 

Overall  Team 

Performance 

12 

.671 

.000** 

Moderately 

Supported 

Question  8  (CO)  TO 

Overall  Team 

Performance 

13 

.664 

.000** 

Moderately 

Supported 

Question  7  (UO)  TO 

Overall  Team 

Performance 

14 

.662 

.000** 

Moderately 

Supported 

Question  8  (CO)  TO 
Question  12 

15 

.658 

.000** 

Moderately 

Supported 

Hypothesis  3b 

Question  7  (UO)  TO 
Question  12 

16 

.643 

.000** 

Moderately 

Supported 

Hypothesis  3a 

Question  10  (CS)  TO 
Question  13 

17 

.640 

.000** 

Moderately 

Supported 

Hypothesis  4b 

Instructor  Performance  TO 
Overall  Team 

Performance 

18 

.630 

.000** 

Moderately 

Supported 

Hypothesis  1 

Question  9  (US)  TO 
Question  13 

19 

.625 

.000** 

Moderately 

Supported 

Hypothesis  4a 

Question  6  (CP)  TO 

Overall  Team 

Performance 

20 

.604 

.000** 

Moderately 

Supported 

Question  6  (CP)  TO 
Question  11 

21 

.594 

.000** 

Moderately 

Supported 

Hypothesis  2b 
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Variables 

Rank 

Pearson’s  r 

p-value 

Results 

Question  5  (UP)  TO 

22 

.513 

.000** 

Moderately 

Question  11 

Supported 

Hypothesis  2a 

Question  5  (UP)  TQ 

Qverall  Team 

23 

.495 

.000** 

Modestly  Supported 

Performance 

Question  10  (CS)  TO 

24 

.486 

.000** 

Modestly  Supported 

Instructor  Performance 

Hypothesis  6f 

Question  7  (UO)  TO 

25 

.466 

.000** 

Modestly  Supported 

Instructor  Performance 

Hypothesis  6c 

Overall  Strategic  Intent  TO 

26 

.463 

.000** 

Modestly  Supported 

Instructor  Performance 

Hypothesis  5 

Question  8  (CO)  TO 

27 

.405 

.002** 

Modestly  Supported 

Instructor  Performance 

Hypothesis  6d 

Question  6  (CP)  TO 

28 

.352 

.007** 

Modestly  Supported 

Instructor  Performance 

Hypothesis  6b 

Question  5  (UP)  TO 

29 

.349 

.008** 

Modestly  Supported 

Instructor  Performance 

Hypothesis  6a 

Question  9  (US)  TO 
Instructor  Performance 

30 

.330 

.012* 

Modestly  Supported 

Hypothesis  6e 

Note:  *  Correlation  is  significant  at  the  .05  level  (two-tailed  test);  **  Correlation  is 
significant  at  the  .01  level  (two-tailed  test). 


The  first  six  entries/cases  in  the  table  above  highlight  the  strongly  supported 
relationship  between  the  six  individual  questions  in  the  team  survey  (Questions  6-10)  and 
overall  team  Strategic  Intent.  These  relationships  are  high  in  magnitude  or  strength 
because  Team  Strategic  Intent  is  defined  by  the  average  of  all  the  team  members’ 
responses  to  the  six  questions  related  to  the  team’s  understanding  and  commitment  to  the 
team’s  purpose  (Questions  5  and  6  respectively),  understanding  and  commitment  to  the 
team’s  objectives  (Questions  7  and  8  respectively),  and  understanding  and  commitment  to 
the  team’s  strategies  (Questions  9  and  10  respectively).  These  results  make  sense  and 
provide  no  real  insight  into  the  research  except  that  Question  8  (Team  Understanding  of 
Qbjectives)  has  the  greatest  strength  of  .921 .  This  indicates  that  this  question  has  the 
greatest  effect  on  overall  Team  Strategic  Intent.  Managers  should  be  aware  that  developing 
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a  strong  sense  of  understanding  of  team  objectives  among  team  members  will  have  the 
most  significant  (largest)  effect  on  the  team’s  overall  Strategic  Intent  perception.  This  can 
also  have  an  effect  on  Team-assessed  and  Instructor-assessed  Team  Performance.  This  is 
consistent  with  the  fact  that  Question  8  has  the  strongest  relationship  to  Question  12  in 
terms  of  comparing  Strategic  Intent  questions  to  their  related  Team  Performance  question. 
“Understanding  the  team  objectives”  as  a  variable  plays  a  major  role  in  both  these 
relationships. 

The  strength  of  the  relationship  between  overall  Team  Strategic  Intent  and  Team 
Performance  (Hypothesis  1 )  at  .731  underscores  the  influence  that  strategic  thinking  or 
developing  clear  and  understandable  strategic  elements  In  a  team  affects  how  the  team  will 
assess  its  performance.  This  is  a  vital  source  of  information  to  educators,  team  and 
business  leaders,  and  team  sponsors/stakeholders.  This  highlights  that  a  team  with  a  clear 
set  of  strategic  characteristics  of  a  team  purpose,  objectives,  and  strategies  will  more 
probably  develop  a  strong  sense  of  being  a  high-performing  team.  Believing  this  will 
empower  the  team  to  greater  team  results  and  even  more  focused  performance.  This 
should  also  produce  better  results  for  the  organizations  that  sponsor  them.  The  leader  of 
this  team  also  needs  to  know  that  a  focused,  intent  team  will  believe  it  will  perform  well. 

Qverall  Team  Strategic  Intent  is  a  key  variable  in  this  study  and  is  analyzed/tested  in 
11  of  the  cases  identified  in  Table  1  above.  The  strength  of  the  relationship  between 
Strategic  Intent  (SI)  and  Team  Performance  at  .731  is  compared  to  the  same  relationship 
between  Strategic  Intent  and  Instructor-assessed  Performance  at  .463.  This  indicates  that 
team  strategic  thinking  has  a  greater  relationship  to  or  effect  upon  Team-assessed  Team 
Performance  than  its  effect  on  Instructor-assessed  Performance.  The  strength  of  team 
Strategic  Intent  on  the  instructors’  assessment  is  significant,  nonetheless,  and  indicates  that 
team  strategic  thinking  not  only  affects  Team  Performance  but  also  how  the  team’s 
instructors  assessed  the  team’s  performance. 

Strategic  Intent  is  a  strong  force  in  or  predictor  of  team  dynamics  and  development. 
Additional  future  studies  should  be  made  to  understand  how  Team  Strategic  Intent  is  related 
to  other  indicators  of  team  success  or  performance,  such  as  quality  of  work,  timeliness, 
problem-solving  effectiveness,  and  overall  team  productivity.  Qverall  Strategic  Intent  (SI) 
has  a  strong  relationship  not  only  to  overall  Team  Performance  at  .731  but  also  when  tested 
against  the  three  questions  that  create  Team  Performance.  The  results  are  the  following: 
724  (Question  1 1  to  SI),  .706  (Question  12  to  SI),  and  .680  (Question  13  to  SI).  This  is  to  be 
expected  and  again  underscores  the  strength  and  value  of  understanding  the  effects  and 
strength  of  Team  Strategic  Intent  on  Team  Performance.  Additional  Pearson’s  r  correlation 
coefficient  tests  highlight  that  when  the  individual  Strategic  Intent  questions  (5-10)  are 
compared  to  the  overall  Team-assessed  Team  Performance,  significant  (.01)  relationships 
occur.  In  fact,  the  results  of  these  tests  are  similar  in  strength  to  the  results  obtained  on  the 
tests  between  the  Strategic  Intent  questions  to  their  related  individual  Team-assessed 
Performance  questions  (11-13). 

Here  are  the  comparisons:  Question  5,  Understanding  Team  Purpose  (Team 
Performance:  .495;  Question  11:  .513);  Question  6,  Commitment  to  Team  Purpose  (Team 
Performance:  .604;  Question  11:  .594);  Question  7,  Understanding  Team  Qbjectives 
(Team  Performance:  .662;  Question  11:  .643);  Question  8,  Commitment  to  Team 
Objectives  (Team  Performance:  .664;  Question  11:  .658);  Question  9,  Understanding 
Team  Strategies  (Team  Performance:  .671;  Question  11:  .625);  and  Question  10, 
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Commitment  to  Team  Strategies  (Team  Performance:  .673;  Question  11:  .640).  Allot 
these  results  have  p-values  of  .000**. 


In  summary,  it  is  concluded  that  these  additional  tests  on  various  Strategic  Intent 
questions  and  overall  Team-assessed  Performance  provided  additional  support  to  the 
previously  conducted  hypotheses  tests.  A  more  robust  test  was  comparing  the  Strategic 
Intent  questions  (5-10)  to  the  related  Team-assessed  Performance  questions  (11-13).  The 
results  moderately  supported  the  direct  relationship  between  these  sets  of  variables.  There 
was  a  moderately  supported  relationship  between  Team  Strategic  Intent  and  Team- 
assessed  Performance  with  all  three  methods:  (a)  overall  Team  Strategic  Intent  to  Overall 
Team  Performance  (Research  Question  1  and  Hypothesis  1),  (b)  results  of  individual 
Strategic  Intent  questions  5-10  to  individual  related  Team  Performance  questions  11-13 
(Research  Questions  2a,  2b,  3a,  3b,  4a,  and  4b),  and  (c)  individual  Strategic  Intent 
questions  5-10  to  overall  Team  Performance  (see  Table  20  results). 

Additional  Findings  on  Demographics  Data 

Additional  tests  were  conducted  on  the  measured  demographic  information  and  its 
relationship  to  overall  team  Strategic  Intent,  Team-assessed  Team  Performance,  and 
Instructor-assessed  Team  Performance.  Twelve  tests  were  conducted,  and  only  3  tests 
were  supported  at  least  the  .05  significance  level.  Two  supported  tests  related  Team 
Educational  Level  to  Team-assessed  Performance  and  to  Instructor-assessed  Team 
Performance.  Pearson’s  rand  Spearman’s  rho  correlation  tests  both  indicated  a  positive 
relationship  between  Team  Educational  Level  and  overall  Team-assessed  Team 
Performance  and  Instructor-assessed  Team  Performance  at  a  .05  significance  level. 

Educational  level  can  make  a  difference  in  Team  Performance,  both  as  assessed  by 
the  team  itself  and  by  the  instructors.  Although  not  significant  at  .05,  there  is  also  a  positive 
effect  on  overall  Team  Strategic  Intent  by  team  Educational  Level.  Although  not  statistically 
significant,  there  does  appear  to  be  some  indication  that  using  teams  is  an  effective  learning 
technique  in  education,  and  business  leaders  employing  teams  in  their  organizations  who 
want  to  enhance  strategic  implementation  of  corporate  strategic  goals  and  initiatives  should 
be  aware  that  teams  with  higher  educational  levels  tend  to  have  higher  Team  Strategic 
Intent  (r  =  .239,  not  significant  at  .05),  higher  overall  Team-assessed  Team  Performance  (r  = 
.296,  p-value  =  .025*),  and  higher  Instructor-assessed  Team  Performance  (r=  .441.  p-value 
=  .001**).  Educational  Level  has  a  positive  effect  on  these  three  research  variables. 
Education  has  a  rather  significant  effect  on  Instructor-assessed  Team  Performance  (r  = 
.441). 

Team  age  and  years  of  experience  have  a  negative  effect  on  Team  Strategic  Intent, 
on  overall  Team-assessed  Team  Performance,  and  on  Instructor-assessed  Team 
Performance.  The  strength  of  the  relationships  is  low,  and  the  significant  levels  are  high. 

No  relationship  was  supported  at  the  .05  significance  level.  Although  not  supported 
statistically  at  an  alpha  of  .05,  this  was  of  interest  to  the  researcher.  Age  and  experience 
have  negative  relationships  to  all  the  research  variables:  Strategic  Intent,  Team-assessed 
Team  Performance,  and  Instructor-assessed  Team  Performance. 

There  is  a  moderately  strong  relationship  between  Team  Experience  and  Team  Age 
(r=  .643,  p-value  =  .001**).  This  is  logical,  and  passed  the  common-sense  test.  The  results 
do  not  affect  this  research  but  highlight  the  strength  of  the  survey  data  to  develop 
conclusions  regarding  the  survey  sample. 
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Conclusions 


The  main  conclusions  in  this  research  are  the  following: 

1 .  There  is  a  statistically  significant  relationship  between  the  overall  team  Strategic 
Intent  and  overall  Team-assessed  Team  Performance.  Teams  that  have  high  overall 
team  Strategic  Intent  (team  purpose,  objectives,  and  strategies)  also  have  high 
overall  Team-assessed  Team  Performance. 

2.  There  is  a  statistically  significant  relationship  between  the  individual  team  Strategic 
Intent  questions  (5-10)  and  overall  Team-assessed  Team  Performance.  Teams  that 
have  high  results  on  individual  team  Strategic  Intent  questions  (5-10)  also  have  high 
results  on  overall  Team-assessed  Team  Performance. 

3.  There  is  a  statistically  significant  relationship  between  the  individual  team  Strategic 
Intent  questions  (5-10)  and  individual  Team-assessed  Team  Performance  questions 
(11-13).  Teams  that  have  high  results  on  individual  team  Strategic  Intent  questions 
(5-10)  also  have  high  results  on  individual  Team-assessed  Team  Performance 
questions  (11-13). 

4.  There  is  a  statistically  significant  relationship  between  the  overall  team  Strategic 
Intent  and  overall  Instructor-assessed  Team  Performance  (Question  4).  Teams  that 
have  high  overall  team  Strategic  Intent  (team  purpose,  objectives,  and  strategies) 
also  have  high  Instructor-assessed  Team  Performance  (Question  4). 

5.  There  is  a  statistically  significant  relationship  between  the  individual  team  Strategic 
Intent  questions  (5-10)  and  individual  Instructor-assessed  Team  Performance 
(Question  4).  Teams  that  possessed  high  scores  on  each  individual  Question  5-10 
dealing  with  team  Strategic  Intent  also  had  high  Instructor-assessed  Team 
Performance. 

6.  There  is  a  statistically  significant  relationship  between  the  overall  Team-assessed 
Team  Performance  (Questions  11-13)  and  overall  Instructor-assessed  Team 
Performance  (Question  4).  Teams  that  have  high  overall  Team-assessed  Team 
Performance  (Questions  11-13)  also  have  high  Instructor-assessed  Team 
Performance  (Question  4). 

7.  There  is  a  statistically  significant  relationship  between  the  overall  Team  Educational 
Level  and  overall  Instructor-assessed  Team  Performance  (Question  4).  Teams  that 
have  high  overall  Team  Educational  Level  also  have  high  Instructor-assessed  Team 
Performance  (Question  4).  There  is  some  indication  (supported  at  .05  significance 
level)  that  there  is  also  a  relationship  between  the  overall  Team  Educational  Level 
and  both  overall  Team-assessed  Team  Performance  (Questions  11-13)  (supported 
at  .05  significance  level)  and  overall  Strategic  Intent  (Questions  5-10)  (not  supported 
at  .05  significance  level). 

8.  There  is  some  indication  (not  supported  at  .05  significance  level)  that  there  is  also  an 
indirect  or  negative  relationship  between  the  overall  Team  Average  Age  and  all  of 
the  following:  (a)  overall  team  Strategic  Intent  (Questions  5-10),  (b)  overall  Team- 
assessed  Team  Performance  (Questions  11-13),  and  (c)  overall  Instructor-assessed 
Team  Performance  (Question  4). 

There  is  some  indication  (not  supported  at  .05  significance  level)  that  there  is  also  an 
indirect  or  negative  between  the  overall  Team  Average  Years  Experience  and  all  the 
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following:  (a)  overall  team  Strategic  Intent  (Questions  5-10),  (b)  overall  Team-assessed 
Team  Performance  (Questions  11-13),  and  (c)  overall  Instructor-assessed  Team 
Performance  (Question  4).  The  strengths  of  these  relationships  and  significance  levels  do 
not  allow  for  statistical  significance  of  these  relationships.  The  interesting  aspect  of  these 
studies  highlight  that  with  more  data  and  research,  age  and  experience  may  have 
statistically  significant  negative  effects  on  the  research  variables  of  overall  team  Strategic 
Intent,  Team-assessed  Team  Performance,  and  Instructor-assessed  Team  Performance. 

Concluding  Statement 

Teams  can  be  a  significant  resource  to  business  leaders  and  lead  to  greater  program 
successes.  Little  empirical  data  exist  on  what  strategic  characteristics  make  teams  more 
effective.  Does  a  work  team’s  success  depend  on  how  strategically  focused  or  intent  the 
team  is?  Do  team-developed  purpose,  objectives,  and  strategies  (strategic  intent)  have  an 
effect  on  how  well  teams  perform?  This  research  study  hypothesized  and  proved  that  work 
team  strategic  intent  characteristics  (team-developed  purpose,  objectives,  and  strategies) 
were  directly  or  positively  related  to  the  performance  of  student  work  teams. 

Significant  positive  correlation  relationships  were  found  in  all  15  studied  hypotheses 
between  work  team  strategic  intent  and  team  performance  as  measured  by  team  self- 
assessments  and  instructor  assessments.  Additionally,  a  positive  correlation  was  found 
between  the  team  self-assessment  of  performance  and  the  instructors’  assessment  of  the 
team  performance. 

The  research  provided  significant  empirical  data  on  the  positive  correlation 
relationships  between  work  team  strategic  intent  and  work  team  performance.  It  also 
defined  the  characteristics  that  were  used  to  determine  the  strategic  intent  of  a  work  team  or 
any  work  unit.  It  created  empirical  support  for  Katzenbach  and  Smith’s  theories  from  their 
studies  in  The  Wisdom  of  Teams:  Creating  the  High-performance  Organization  (2003)  on 
the  success  of  real  teams,  based  on  being  committed  to  a  common  purpose  and 
performance  goals.  Additionally,  it  created  a  survey  to  measure  the  strategic  intent  of  team 
members  and  teams  in  general.  Finally,  it  introduced  the  study  of  strategic  thinking  or  use 
of  strategic  intent  as  a  method  or  process  for  evaluating  team  performance. 

The  complexity  of  team  performance  and  the  large  number  of  future  potential 
influences  and  additional  areas  of  research  needed  on  teams  were  highlighted  in  the 
research.  This  may  help  explain  why  so  many  organizations  using  teams  in  both  the  public 
and  private  sector  today  are  having  difficulty  as  they  try  to  reposition  themselves  in  an  ever 
more  turbulent  environment  and  why  teams  are  often  not  as  effective  or  successful  as 
possible. 

Properly  disciplined,  focused,  and  integrated  teams  are  the  ones  that  become  high- 
performing  teams,  and  are  considered  “the  most  versatile  unit  organizations  have  for 
meeting  both  performance  and  challenges  in  today’s  complex  world”  (Katzenbach  &  Smith, 
2003,  p.  xiii).  This  study  has  identified  that  Strategic  Intent  or  clearly  focused  team  purpose, 
objectives,  and  strategies  can  make  teams  more  high-performing  and  even  more  versatile 
and  effective  in  an  organization — both  in  the  short  and  long  term. 
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Appendix  A.  Team  Survey 

WORK  TEAM 

INTENT  AND  WORK  TEAM 
PERFORMANCE 

This  is  a  university  doctorate-level  and  Defense  Acquisition  University 
co-sponsored  survey.  It  will  contribute  to  advancing  the  body  of  knowledge  to 
help  identify  if  and  how  much  the  alignment  of  work  team  strategic  intent  (which 
relates  to  work  team  purpose,  objectives,  and  strategies)  contribute  to  work 
team  performance.  This  information  will  help  enhance  work  team  performance 
effectiveness  in  DAU  classrooms  and  in  DoD  organizations. 
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The  information  gathered  in  this  questionnaire  is  confidential.  No  specific  response  will 
be  shared  with  any  respondent.  Only  aggregate  information  will  be  discussed  in  this  study. 

No  specific  information  about  you  personaily  or  your  team  will  be  released. 

The  survey  is  optional.  If  you  do  not  want  to  complete  the  survey  inform  the  researcher  or 

instructor  that  is  administering  the  survey. 

Consent:  Your  completion  of  this  survey  and  submittal  to  the  researchers  provide  your 
consent  to  the  researcher  to  use  the  data  in  the  survey  to  conduct  anaiyses  and  determine 
the  results  and  conclusions  related  to  their  research. 

Additional  Information.  Completed  Surveys,  and  Requests 

for  Results  of  this  Research  Should  be  Sent  to: 

Tom  Edison 

Alliant  International  University 
536  H  Ave 

Coronado,  Calif.  92118 
Tel:  619-437-4123  (Home) 

619-524-4815  (Work) 
tom.edison@dau.mil 

Thank  you  very  much  for  your  cooperation! 


Work  Team  Information 

_ Please  fill  in  the  blanks  below: 

1 .  Name  and/or  number  assigned  to  your  work  team: 


2.  Date  (month/day/yr)  your  work  team  began  its  work  (date  class  began)  and  ended: 
class  began  from _ and  lasted  to _ . 

3.  Specific  stated  purpose/charter  of  your  work  team: 


4.  Location  of  your  class/team: 


Questions  5  to  10  that  follow  ask  for  your  judgment  or  perception  on  how 
similar  or  aligned  your  understanding  of  and/or  commitment  to  specific  work  team 
elements  (purpose,  objectives,  and  strategies)  are  to  those  of  other  members  on 

your  work  team. 

Purpose 

Questions  5  to  6:  Team’s  purpose  refers  to  your  work  team’s  overall  goal  for  the  future  of  the  work 
team  during  next  six  weeks.  Consider  this  as  your  intent  or  focus  that  collectively  provided  the  work 
team  a  future  goal  for  the  team’s  activities  and  affected  the  team’s  decision  making  and  performance. 


5.  How  similar  is  your  understanding  of  your  work  team’s  purpose  to  that  of  the 
other  members  on  your  work  team?  (Circle  correct  rating.) 
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Your 

Understanding  of 
Work  Team’s 
Purpose 

Not  Similar 
at  all 

Somewhat 

Similar 

Very 

Similar 

Extremely 

Similar 

1 

2 

3 

4 

6.  How  simi 


ar  is  your  commitment  to  your  work  team’s  purpose  to  that  of  the  other 


members  on  your  work  team?  (Circle  correct  rating.) 


Your  Commitment 
to  Work  Team’s 
Purpose 

Not  Similar 
at  all 

Somewhat 

Similar 

Very 

Similar 

Extremely 

Similar 

1 

2 

3 

4 

Objectives 

Questions  7  to  8:  Team’s  objectives  are  the  specific  work  team  performance  targets  or 
shorter-term  targets  your  work  team  was  formed  for  and  is  working  to  accompiish. 

7.  How  similar  is  your  understanding  of  your  work  team’s  objectives  to  that  of  the 
other  members  on  your  work  team?  (Circle  correct  rating.) 


Your 

Understanding  of 
Team’s  Objectives 

Not  Similar 
at  all 

Somewhat 

Similar 

Very 

Similar 

Extremely 

Similar 

1 

2 

3 

4 

8.  How  simi 

ar  is  vour  commitment  to  vour  work  team’s  objectives  to  that  of  the 

other  members  on  your  work  team?  (Circle  correct  rating.) 


Your  Commitment 
to  Team’s 
Objectives 

Not  Similar 
at  all 

Somewhat 

Similar 

Very 

Similar 

Extremely 

Similar 

1 

2 

3 

4 

Strategies 


Questions  9  to  10:  Team’s  strategies  are  work  team  procedures,  plans,  approach, 
and  methods  used  to  achieve  the  overall  work  team’s  purpose  and  objectives. 


9.  How  similar  is  your  understanding  of  your  work  team’s  strategies  to  that  of  the 
other  members  on  your  work  team?  (Circle  correct  rating.) 


Your 

Understanding  of 
Team’s 
Strategies 

Not  Similar 
at  all 

Somewhat 

Similar 

Very 

Similar 

Extremely 

Similar 

1 

2 

3 

4 

1 0.  How  similar  is  your  commitment  to  your  work  team’s  strategies  to  that  of  the 
other  members  on  your  work  team?  (Circle  correct  rating.) 
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Your 

Commitment  to 
Team’s 
Strategies 

Not  Similar 
at  all 

Somewhat 

Similar 

Very 

Similar 

Extremely 

Similar 

1 

2 

3 

4 

TEAM  PERFORM 

ANCE 

Questions  11-13:  Team  performance  relates  directly  to  your  team’s 
accomplishment  of  the  team’s  purpose  (evaluated  in  questions  5-6)  and  objectives 
(evaluated  in  questions  7-8)  using  the  team  strategies  (evaluated  in  questions  9- 

10). 


1 1 .  Please  indicate  how  you  rate  your  work  team's  overall  performance  in  terms  of 
accomplishing  your  work  team’s  purpose.  (Circle  one  rating  number  that  represents  the 
success  of  team  accomplishments  in  relation  to  your  team  purpose.) 

Poor  Below  Ave  Average  Good  Excellent 

1  2  3  4  5 


12.  Please  indicate  how  you  rate  your  work  team's  overall  performance  in  terms  of 
accomplishing  your  work  team’s  objectives.  (Circle  one  rating  number  that  represents  the 
success  of  team  accomplishments  in  relation  to  your  team  objectives.) 

Poor  Below  Ave  Average  Good  Excellent 

1  2  3  4  5 


13.  Please  indicate  how  you  rate  your  work  team's  overall  performance  in  terms  of 
using  your  work  team’s  strategies.  (Circle  one  rating  number  that  represents  the  success  of 
team  accomplishments  in  relation  to  using  your  team  strategies.) 


Poor 

Below  Ave 

Average 

Good 

Excellent 

1 

2 

3 

4 

5 

Personal  Information 


(This  information  will  be  held  in  strictest  confidence.  Please  fill  in  the  blanks  below.) 

14.  In  the  following  questions  please  provide  some  personal  information  about  yourself: 

a.  Total  years  work  experience; _ years  (Fill  in  the  blank) 

b.  Gender:  Male  Female  (Circle  correct  answer) 
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c. 


Your  age: _ (Fill  in  the  blank) 

d.  Current  work  status; 

1 .  If  active  duty  military  answer  below:  (Circle  answer  or  check  in  box) 

In  what  Service  were  you  on  active  duty  on  January  2,  2006? 

Army 

Navy 

Marine  Corps 
Air  Force 


2.  If  government  civilian  answer  below:  (Circle  answer  or  check  in  box) 

For  which  Department  of  Defense  (DoD)  component  did  you  work  on  January  2,  2006? 

Army 

Navy 

Marine  Corps 
Air  Force 

DoD  Agency  or  Activity 


3.  If  civilian  contractor  answer  below:  (Fill  in  blank) 

What  company  do  you  work  for? _ 

f.  Work/functional  background  that  you’ve  worked  more  than  50%  of  your  total  career 
time.  (Check  one  most  correct) 

_ Engineering  _ Procurement 

_ Logistics  _ Provisioning/Supply 

_ Sales  _ Financial  Management 

_ Marketing  _ Information  Technology 

_ Quality  Assurance  _ Software  Management 

_ Program  Management  _ Other  (Write  in  Work 

Operations  Background  greater  than  50%) 

_ Contracting 
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g.  Highest  educational  level  completed;  (Circle  most  correct  choice) 


1 

Some  High  School 

6 

Some  Post-Graduate 

Courses 

2 

High  School  Graduate 

7 

Masters  Degree 

3 

Some  College  (1-2  yrs) 

8 

Some  Post-Masters  Courses 

4 

Some  College  (3-4  yrs) 

9 

PhD/Doctorate  Degree 

5 

Bachelors,  College  Graduate 

10 

Other 

Thank  you  very  much  for  your  help  with  this  research  project! 


Comments  or  Recommendations: 

Appendix  B.  Overall  Test  Results 


1.  Correlation  Results  of  Overall  Strategic  Intent  and  Overall  Team-assessed 
Performance — (RQ  1) 


Correlations 


a 

TeamPerf 

a 

Pearson  Correlation 

1 

.731** 

Sig.  (2-tailed) 

.000 

N 

57 

57 

TeamPerf 

Pearson  Correlation 

.731** 

1 

Sig.  (2-tailed) 

.000 

N 

57 

57 

**•  Correlation  is  significant  at  the  0.01  level  (2-tailed). 


2.  Correlation  Results  of  Strategic  Intent  Elements  (Q  5-10)  to 

Work  Team-assessed  Performance  (Q  11-13) — (RQ  2a-b,  3a-b  &  4a-b): 
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Descriptive  Statistics 


Mean 

Std.  Deviation 

N 

Q5TUP 

3.2120 

.33203 

57 

Q6TCP 

3.2104 

.32512 

57 

Q7TUO 

3.2141 

.32723 

57 

Q8TCO 

3.1840 

.37275 

57 

Q9TUS 

3.0389 

.34556 

57 

Q10TCS 

3.1209 

.36104 

57 

Q11PP 

4.3788 

.45923 

57 

Q12PO 

4.3041 

.47522 

57 

Q13PS 

4.1885 

.47659 

57 

Correlations 


Q5TUP 

Q6TCP 

Q7TUO 

Q8TCO 

Q9TUS 

310TCS 

Q11PP 

Q12PO 

Q13PS 

Q5TUF  Pearson  Corre 

1 

.652* 

.704* 

.675* 

.564* 

.525* 

.513* 

.452* 

.464* 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

N 

57 

57 

57 

57 

57 

57 

57 

57 

57 

Q6TCF  Pearson  Corre 

.652* 

1 

.649* 

.860* 

.587* 

.756* 

.594* 

.562* 

.589* 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

N 

57 

57 

57 

57 

57 

57 

57 

57 

57 

Q7TUC  Pearson  Corre 

.704* 

.649* 

1 

.786* 

.699* 

.718* 

.663* 

.643* 

.608* 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

N 

57 

57 

57 

57 

57 

57 

57 

57 

57 

Q8TCC  Pearson  Corre 

.675* 

.860* 

.786* 

1 

.622* 

.803* 

.677* 

.658* 

.585* 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

N 

57 

57 

57 

57 

57 

57 

57 

57 

57 

Q9TUS  Pearson  Corre 

.564* 

.587* 

.699* 

.622* 

1 

.747* 

.635* 

.677* 

.625* 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

N 

57 

57 

57 

57 

57 

57 

57 

57 

57 

Q10TC  Pearson  Corre 

.525* 

.756* 

.718* 

.803* 

.747* 

1 

.655* 

.648* 

.640* 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

N 

57 

57 

57 

57 

57 

57 

57 

57 

57 

Q1 1 PP  Pearson  Corre 

.513* 

.594* 

.663* 

.677* 

.635* 

.655* 

1 

.928* 

.877* 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

N 

57 

57 

57 

57 

57 

57 

57 

57 

57 

Q12PC  Pearson  Corre 

.452* 

.562* 

.643* 

.658* 

.677* 

.648* 

.928* 

1 

.864* 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

N 

57 

57 

57 

57 

57 

57 

57 

57 

57 

Q13PS  Pearson  Corre 

.464* 

.589* 

.608* 

.585* 

.625* 

.640* 

.877* 

.864* 

1 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

N 

57 

57 

57 

57 

57 

57 

57 

57 

57 

**-Correlation  is  significant  at  the  0.01  level  (2-tailed). 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EORINEORMED  CHANGE 


-118- 


3.  Correlation  Results  of  Overall  Strategic  Intent  and  Overall  Instructor- 
assessed  Performance — (RQ  5): 


Correlations 


a 

Q14lnstrPerf 

a 

Pearson  Correlation 

1 

.463** 

Sig.  (2-taile(d) 

.000 

N 

57 

57 

Q14lnstrPerf 

Pearson  Correlation 

.463** 

1 

Sig.  (2-taile(d) 

.000 

N 

57 

57 

Correlation  is  significant  at  the  0.01  level  (2-tailed). 


4.  Correlation  Results  of  Strategic  Intent  Elements  (Q5-10)  to 
Instructor  Assessed  Performance — (RQ  6a-f): 


Correlations 


Q5TUP 

Q6TCP 

Q7TUO 

Q8TCO 

Q9TUS 

Q10TCS 

Q14lnstrPerf 

Q5TUP 

Pearson  Correlatior 

1 

.652*” 

.704*” 

.675*” 

.564*” 

.525*” 

.349*” 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.008 

N 

57 

57 

57 

57 

57 

57 

57 

Q6TCP 

Pearson  Correlatior 

.652*^ 

1 

.649*” 

.860*” 

.587*” 

.756*” 

.352*” 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.007 

N 

57 

57 

57 

57 

57 

57 

57 

Q7TUO 

Pearson  Correlatior 

.704*’" 

.649*” 

1 

.786*” 

.699*” 

.718*” 

.466*” 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

N 

57 

57 

57 

57 

57 

57 

57 

Q8TCO 

Pearson  Correlatior 

.675*” 

.860*” 

.786*” 

1 

.622*” 

.803*” 

.405*” 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.002 

N 

57 

57 

57 

57 

57 

57 

57 

Q9TUS 

Pearson  Correlatior 

.564*” 

.587*” 

.699*” 

.622*” 

1 

.747*” 
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Sig.  (2-tailed) 

.000 

.000 
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.000 

.000 

.012 

N 

57 

57 

57 

57 

57 

57 

57 

Q10TCS 

Pearson  Correlatior 

.525*” 

.756*” 

.718*” 

.803*” 

.747*” 

1 

.486*” 

Sig.  (2-tailed) 

.000 

.000 

.000 

.000 

.000 

.000 

N 

57 

57 

57 

57 

57 

57 

57 
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.349*” 

.352*” 

.466*” 

.405*” 

.330* 

.486*” 

1 

Sig.  (2-tailed) 

.008 

.007 

.000 

.002 

.012 

.000 

N 

57 

57 

57 

57 

57 

57 

57 

**■  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
*■  Correlation  is  significant  at  the  0.05  level  (2-tailed). 
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5.  Correlation  Results  of  Overall  Team -assessed  Performance  and  Overall  Instructor 
Assessed  Performance — (RQ  7): 


Correlations 


TeamPerf 

Q14lnstrPerf 

TeamPerf 

Pearson  Correlation 

1 

.630** 

Sig.  (2-tailed) 

.000 

N 

57 

57 

Q14lnstrPerf 

Pearson  Correlation 

.630** 

1 

Sig.  (2-tailed) 

.000 

N 

57 

57 

Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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AEGIS  and  Ship  Self-defense  System  (SSDS)  Platforms: 
Using  KVA  Analysis,  Risk  Simulation  and  Strategic  Real 
Options  to  Assess  Operational  Effectiveness 


Presenter:  Dr.  Thomas  Housel,  PhD,  specializes  in  valuing  intellectuai  capitai,  telecommunications, 
information  technoiogy,  value-based  business  process  reengineering,  and  knowledge  value 
measurement  in  profit  and  non-profit  organizations.  He  is  currently  a  tenured  Full  Professor  for  the 
Information  Sciences  (Systems)  Department.  His  current  research  focuses  on  the  use  of  “Real 
Options”  models  in  identifying,  valuing,  maintaining,  and  exercising  options  in  military  decision¬ 
making.  Prior  to  joining  NPS,  he  also  was  a  Research  Fellow  for  the  Center  for  Telecommunications 
Management  and  Associate  Professor  at  the  Marshall  School  of  Business  at  the  University  of 
Southern  California.  Housel  has  been  the  Chief  Business  Process  Engineer  for  Pacific  Bell,  where  he 
completed  numerous  reengineering  projects  and  developed  a  new  objective  method  for  objectively 
measuring  the  value-added  by  reengineering.  His  last  assignment  in  the  corporate  world  was  as  the 
Chief  of  Consumer  Market  Research  for  Telecom  Italia  in  Venice,  Italy,  where  he  developed  new 
methods  for  predicting  the  adoption  rates  for  new  interactive  multimedia  broadband  applications.  He 
is  Managing  Partner  for  Business  Process  Auditors,  a  firm  that  specializes  in  training  Big  Six 
consultants,  large  manufacturing  and  service  companies  in  the  Knowledge  Value-Added  methodology 
for  objectively  measuring  the  return  generated  by  corporate  knowledge  assets/intellectual  capital. 

He  received  his  PhD  from  the  University  of  Utah  in  1980.  He  won  the  prestigious  Society  for 
Information  Management  award  for  best  paper  in  the  field  in  1986.  His  work  on  measuring  the  value 
of  intellectual  capital  has  been  featured  in  a  Fortune  cover  story  (October  3,  1994)  and  Investor’s 
Business  Daily,  numerous  books,  professional  periodicals,  and  academic  journals  (most  recently  in 
the  Journal  of  Intellectual  Capital,  vol  2.,  2005).  His  latest  books  include:  Measuring  and  Managing 
Knowledge  and  Global  Telecommunications  Revolution:  The  Business  Perspective  with  McGraw-Hill 
(both  in  2001). 

Presenter:  Dr.  Johnathan  Mun,  PhD,  MBA,  MS,  CFG,  CRA,  FRM,  MIFC.  Dr.  Johnathan  C.  Mun  is  a 
Research  Professor  at  the  Naval  Post  Graduate  School  and  is  the  CEO  of  Real  Options  Valuation 
LLC,  a  consulting,  training,  and  software  development  firm  specializing  in  real  options,  employee 
stock  options,  financial  valuation,  and  risk  analysis  located  in  Northern  California.  He  is  the  creator  of 
the  Real  Option  Super  Lattice  So/ver  software,  Monte  Carlo  Risk  Simulator  software,  and  Employee 
Stock  Options  Valuation  software  at  the  firm.  The  Employee  Stock  Options  Valuation  software  was 
used  by  the  Financial  Accounting  Standards  Board  (FASB)  to  develop  their  example  valuation  (A87) 
in  the  2004  FAS  123  requirements  (he  has  also  advised  the  Board  of  Directors  at  FASB  on  several 
occasions  on  binomial  and  options  valuation).  He  has  authored  numerous  books  including  Real 
Options  Analysis:  Tools  and  Techniques  (Wiley  2002,  with  a  second  edition  forthcoming  September 
2005),  Real  Options  Analysis  Course  (Wiley  2003),  Applied  Risk  Analysis  (Wiley  2003),  and  Valuing 
Employee  Stock  Options  (Wiley  2004).  His  books  and  software  are  being  used  around  the  world  at 
top  universities  (including  the  Bern  Institute  in  Germany,  Chung-Ang  University  in  South  Korea, 
Georgetown  University,  ITESM  in  Mexico,  Massachusetts  Institute  of  Technology,  New  York 
University,  Stockholm  University  in  Sweden,  University  of  the  Andes  in  Chile,  University  of  Chile, 
University  of  Pennsylvania  Wharton  School,  University  of  York  in  the  United  Kingdom,  and  Edinburgh 
University  in  Scotland,  among  others. 

He  has  been  a  finance  and  economics  professor  and  has  taught  courses  in  financial  management, 
investments,  real  options,  economics,  and  statistics  at  the  undergraduate  and  the  graduate  MBA 
levels.  He  has  taught  at  universities  all  over  the  world,  from  the  University  of  Applied  Sciences 
(Switzerland  and  Germany)  and  St.  Mary’s  College  (California),  and  has  chaired  many  graduate 
research  thesis  committees.  He  was  formerly  the  Vice  President  of  Analytics  at  Decisioneering,  Inc., 
where  he  headed  up  the  development  of  real  options  and  financial  analytics  software  products, 
analytical  consulting,  training,  and  technical  support,  and  where  he  was  the  creator  of  the  Real 
Options  Analysis  Toolkit  software,  the  predecessor  of  the  Super  Lattice  Software  discussed  in  this 
book.  Prior  to  joining  Decisioneering,  he  was  a  Consulting  Manager  and  Financial  Economist  in  the 
Valuation  Services  and  Global  Financial  Services  practice  of  KPMG  Consulting  and  a  Manager  with 
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the  Economic  Consulting  Services  practice  at  KPMG  LLP.  He  has  extensive  experience  in 
econometric  modeling,  financial  analysis,  real  options,  economic  analysis,  and  statistics. 

He  has  most  recently  been  working  on  Department  of  Defense  (DoD)  projects  that  apply  real 
options — integrated  risk  management  to  support  analysis  of  alternatives,  strategic  courses  of  action, 
and  acquisitions  programs.  He  has,  with  Dr.  Tom  Housel  of  the  Naval  Post  Graduate  School, 
developed  a  5-day  intensive  training  course  to  teach  DoD  analysts  and  managers  how  to  apply  the 
real  options  analysis  technique  to  various  decision  making  problems.  He  has  also  developed  a 
module  for  the  Homeland  Defense  technology  class  for  the  masters  program  at  the  Naval  Post 
Graduate  School. 

Dr.  Mun  received  his  PhD  in  Finance  and  Economics  from  Lehigh  University,  where  his  research  and 
academic  interests  were  in  the  areas  of  Investment  Finance,  Econometric  Modeling,  Financial 
Options,  Corporate  Finance,  and  Microeconomic  Theory.  He  also  has  a  MBA  in  business 
administration,  a  MS  in  management  science,  and  a  BS  in  Biology  and  Physics.  He  is  Certified  in 
Financial  Risk  Management  (FRM),  Certified  in  Financial  Consulting  (CFC),  and  is  Certified  in  Risk 
Analysis  (CRA).  He  is  a  member  of  the  American  Mensa,  Phi  Beta  Kappa  Honor  Society,  and  Golden 
Key  Honor  Society  as  well  as  several  other  professional  organizations,  including  the  Eastern  and 
Southern  Finance  Associations,  American  Economic  Association,  and  Global  Association  of  Risk 
Professionals.  Finally,  he  has  written  many  academic  articles  published  in  the  Journal  of  the 
Advances  in  Quantitative  Accounting  and  Finance,  the  Global  Finance  Journal,  the  International 
Financial  Review,  the  Journal  of  Financial  Analysis,  the  Journai  of  Applied  Financial  Economics,  the 
Journai  of  Internationai  Financiai  Markets,  Institutions  and  Money,  the  Financiai  Engineering  News, 
and  the  Journal  of  the  Society  of  Petroleum  Engineers. 

Presenter:  Eric  Tarantino,  is  a  recent  graduate  of  the  University  of  Virginia.  He  graduated  as  a 
Bachelor  of  Arts  with  High  Distinction,  receiving  majors  in  both  Anthropology  and  Economics. 
Currently  he  works  as  a  research  associate  at  the  Naval  Postgraduate  School  where  he  focuses  on 
valuation  studies  for  the  Department  of  Defense  (DoD).  After  receiving  his  certification  in  Risk 
Management  (CRM)  from  the  International  Institute  of  Professional  Education  and  Research  (IIPER), 
Eric  has  aided  in  many  studies  including:  assessing  the  return  on  investment  of  various  options  for 
upgrading  the  AEGIS  weapons  system;  analyzing  the  SHIPMAIN  program  and  projecting  its  potential 
value  added  when  three  dimensional  visualization  and  product  lifecycle  management  collaborative 
tools  are  used;  an  efficiency  study  of  the  Army’s  Rapid  Equipping  Force,  rapid  acquisition  process; 
and  will  soon  be  a  coauthor  of  a  book  chapter  discussing  the  SHIPYARD  Planning  Process.  His  work 
focuses  on  the  use  of  the  Knowledge  Value  Added  methodology  coupled  with  Real  Options  analysis. 
Eric  is  also  training  to  become  a  professional  Scuba  Diver. 
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Abstract 

Modern,  analytical  tools  are  critical  to  understand  the  impact  of  open  architecture 
technology  and  open  business  models  on  naval  warfighting  processes  and  procedures. 
These  tools  must  measure  the  operational  value  of  a  system  from  an  end-user,  warfighter 
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perspective,  identify  areas  of  deficiencies  in  capabilities,  and  flag  areas  for  potential 
acquisitions.  One  advantage  of  examining  open  architected  system  upgrade  options  from  a 
warfighter  perspective  is  that  the  new  systems  can  be  integrated  with  reengineered 
processes  more  easily  leading  to  improved  process  performance.  This  perspective,  using 
OA  to  upgrade  existing  IWS  systems,  ensures  that  upgrades  will  lead  to  improved 
warfighting  capabilities.  Traditional  measurement  tools  used  for  cost  analysis  cannot 
calculate  the  total  value  of  upgrading  a  system  to  support  an  improved  warfighting 
capability,  particularly  the  improved  operational  value  resulting  from  reengineering  of 
warfighting  processes. 

The  Knowledge  Value  Added/Real  Options  (KVA+RO)  Valuation  Framework  is  a  tool 
designed  to  assist  decision-makers  in  making  technology  acquisitions.  This  paper  describes 
research  using  the  KVA+RO  framework  for  estimating  return  on  investment,  in  an  open 
architecture  approach,  to  upgrading  and/or  replacing  aging  IWS  AEGIS  and  SSDS  systems. 
The  results  of  the  research  indicated  that  using  the  open  architecture  (OA)model,  in 
combination  with  the  “leave  and  layer”  approach,  was  approximately  five  times  more 
valuable  than  the  current  proprietary  approach  to  system  replacement  and  was 
approximately  twice  as  valuable  as  a  complete  retrofit  and  replace  strategy.  “Leave  and 
layer”  provided  the  highest  return  on  investment  for  replacing  the  AEGIS  system  with  the 
lowest  risk.  The  ultimate  success  of  the  OA  approach  is  dependent  on  the  ability  of  the 
multiple  parties  to  system  development  and  deployment  to  collaborate.  Collaboration,  along 
with  the  tools  that  facilitate  collaboration,  is  critical  to  the  success  of  any  of  the  OA 
approaches. 

Keywords:  Return  on  Investment,  Real  Options,  AEGIS,  SSDS,  Integrated  Risk 
Management 
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Executive  Summary 

The  US  Navy  (Navy)  is  transforming  traditional  business  practices  through  Naval 
Open  Architecture  (Naval  OA).  Naval  OA,  a  multi-faceted,  enterprise-wide  business  model 
and  product-line  strategy  leverages  “open”  computer  design  principles  and  architectures.  It 
expands  the  technological  open  architecture  (OA)  model  and  taps  into  a  multiple-developer 
network  to  deliver  cost-effective,  innovative,  and  rapid/spiral  acquisition  capabilities.  In  the 
migration  to  an  OA  business  model,  billions  of  dollars  in  software  and  hardware 
development  expenditures,  along  with  subsequent  maintenance  costs,  are  at  stake. 

PEO  IWS  tasked  a  research  team  from  the  Naval  Postgraduate  School  (NPS)  to 
develop  a  methodology  for  estimating  return  on  investment  (ROI)  using  an  OA  approach  to 
upgrading  and/or  replacing  the  aging  Integrated  Weapons  Systems  (IWS)  AEGIS  and 
SSDS  systems.  The  methodology  also  had  to  be  capable  of  estimating  total  value  of 
strategic  alternative  options  for  replacing  existing  AEGIS  functionality. 

Approaching  the  project  from  a  customer-based,  warfighter  perspective,  the  NPS 
team  applied  the  Knowledge  Value  Added/Real  Options  (KVA+RO)  valuation/risk  portfolio 
management  framework  to  reengineering  situational  awareness  (SA)  procedures  used  in 
the  AEGIS  and  SSDS  platforms.^  Track  management  sub-processes  used  in  SA 
procedures  were  analyzed  through  the  KVA  process  reengineering  methodology  under  “As 
Is,”  “To  Be,”  “Radical  1,”  and  Radical  2”  scenarios.  ROI  metrics  on  individual  sub-processes 
and  watch  stations  for  AEGIS  and  SSDS  were  generated  through  KVA,  with  a  particular 
focus  on  systems  interoperability.  ROI  estimates  reached  as  high  as  404%  for  AEGIS  and 
399%  for  SSDS. 

Real  options  analysis  was  then  performed  to  determine  the  prospective  value  of 
upgrading  the  AEGIS  IWS  over  a  nine-year  period  from  KVA  data  inputs.  Three  options  of 
“Strategy  A:  As  Is”  (i.e.,  maintain  the  existing  proprietary  approach),  “Strategy  B:  DDX  OA — 
Develop  and  Retrofit”  (i.e.,  develop  a  complete  system  using  an  OA  approach  and  replace 
the  existing  AEGIS  system),  and  “Strategy  C:  Aegis  OA — Leave  and  Layer”  (i.e.,  use  an  OA 
approach  and  replace  AEGIS  modules  overtime)  represent  potential  system  development 
and  deplolyment  strategies;  each  a  unique  path  with  risks  and  benefits.  Real  Options 
values  ranged  from  $12  billion  to  $58.8  billion  for  the  strategic  choices. 


The  KVA+RO  Framework 

KVA+RO  is  a  comprehensive  measurement  process  and  an  integrated  tool  set  that 
defines,  measures  and  evaluates  the  total  value  of  given  IWS  acquisitions.  It  captures  data 
across  a  spectrum  of  organizations  to  compare  returns  on  investments,  outputs,  processes, 
capabilities,  risks,  strategic  alternatives,  costs,  and  value  (i.e.,  comparable  revenue). 
KVA+RO  analytically  quantifies  uncertainty  and  risks  elements  inherent  in  predicting  the 
future,  includes  ways  to  mitigate  these  risks  through  strategic  options  with  analysis  of 
alternatives,  and  by  analytically  developing  and  allocating  budgets  to  optimize  project 
portfolios. 


^  Although  the  total  functionalities  of  AEGiS  and  SSDS  IWS  systems  are  so  broad,  we  focused  on  situational  awareness 
because  it  is  the  most  promising  area  for  upgrading  and  reengineering. 
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Knowledge-based  Metrics:  Knowledge  Value  Added  (KVA) 

KVA  measures  the  value  provided  by  human  capital  assets  and  IT  assets  by  an 
organization,  process  or  function  at  the  sub-process  level.  Using  a  “market  comparables” 
valuation  technique,  it  monetizes  the  outputs  of  all  assets,  including  intangible  knowledge 
assets.  Using  market  comparables  provides  a  means  for  valuing  the  outputs  of  warfighting 
processes  in  the  common  units  of  money.  This,  in  turn,  makes  it  possible  to  use  powerful 
financial  metrics  in  forecasting  the  value  of  various  strategic  options  for  replacing  aging  IWS 
systems. 

Capturing  the  value  embedded  in  an  organization’s  core  processes,  employees  and 
IT  enables  the  actual  cost  and  revenue  of  a  product  or  service  to  be  calculated.  Analyses 
like  ROI  on  individual  projects,  programs,  processes  and  sub-processes  within  a  portfolio  of 
IT  acquisitions  can  be  derived  through  the  KVA  methodology. 

Risk  Analysis:  Real  Options  (RO) 

Potential  strategic  investments  can  then  be  evaluated  with  real  options  analysis 
based  on  KVA  data.  The  analysis  applied  is  a  robust  and  analytical  process  incorporating 
the  risk  identification  (applying  various  sensitivity  techniques),  risk  quantification  (applying 
Monte  Carlo  simulation),  risk  valuation  (applying  real  options  analysis),  risk  mitigation 
(utilizing  real  options  framing),  and  risk  diversification  (employing  analytical  portfolio 
optimization). 

Study  Results  and  Recommendations  Summary 

The  results  of  our  analysis  include: 

•  The  KVA+RO  valuation  framework,  a  viable  methodology  for  estimating 
ROIs  and  projecting  valuation  of  acquisition  options,  should  be  used 
across  the  board.  Several  Department  of  Defense  projects  are  implementing 
the  framework.  The  methodology  also  supports  the  CNO’s  recent  directive  of 
accelerating  adoption  of  open-business  models  and  providing  a  methodology 
to  assess  the  business  risks  and  benefits  of  various  OA-based  acquisition 
strategies. 

•  Upgrading  existing  IWS  functionality  to  support  reengineering  elements  of 
existing  track-management  process  appears  beneficial.  ROIs  ranged  from 
212%  to  404%  for  the  AEGIS  platform  and  ROIs  for  the  SSDS  platform  were 
also  significant.  ROI  results  are  shown  in  Figures  1  and  2.®  In  addition. 
Tables  1  and  2  provide  a  detailed  analysis  of  the  ROIs  for  reengineering  the 
track  management  process.  Table  3  summarizes  the  reengineered 
processes  and  subsequent  benefits.  The  results  are  based  on  the 
assumption  that  the  IWS  systems  could  be  developed  within  an  OA 
framework  to  support  the  reengineered  process  designs. 


®  Radical  1  scenario  assumes  the  improvements  of  the  “To  Be”  scenario  while  the  Radical  2  scenario  assumes  cumuiative 
improvements  from  all  three  scenarios. 
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Figure  1.  KVA  Results:  AEGIS  ROI  Estimates 


Figure  2.  KVA  Results:  SSDS  ROI  Estimates 


Tables  1  and  2  are  more  detailed  results  for  ROI  analysis  for  “As  Is”  and  the  other  three 
increasingly  automated  scenarios. 
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AS  IS 

TO  BE 

RAD  1 

RAD  2 

CORRELATE 

Obtain  Link  Information 

3421% 

3307% 

3063% 

2633% 

Identify  "Same  Contact,  Multiple  Track" 

-91% 

-91% 

2061% 

1756% 

Verify  Other  Track  Sources 

-95% 

-95% 

-95% 

-96% 

Correlate  sub-total 

1184% 

1141% 

1506% 

1296% 

TRACK 

Monitor  Suspect  Tracks 

-98% 

-99% 

-99% 

-99% 

Update  Tracks 

-97% 

-97% 

361% 

310% 

Update  GCCS-M 

-97% 

91% 

84% 

69% 

Track  sub-total 

-98% 

-94% 

-58% 

-64% 

IDENTIFY 

Verify  IFF  signal 

802% 

769% 

706% 

607% 

Verify  EW  emissions 

-91% 

-91% 

-92% 

509% 

Verify  Point  of  Origin 

-98% 

4121% 

3821% 

3332% 

Match  Against  ATO 

-98% 

4206% 

3890% 

3382% 

Match  Against  CommAir  Profile 

863% 

835% 

763% 

643% 

Match  Against  Intel  Information 

-97% 

-97% 

-97% 

3814% 

Examine  Kinematic  Data 

-96% 

-96% 

-97% 

-97% 

Obtain  Visual  ID 

-100% 

-100% 

-100% 

-100% 

Conduct  Verbal  Query 

-99% 

-99% 

-99% 

-99% 

Identify  sub-total 

8% 

60% 

50% 

326% 

RELAY 

Send  Over  Links 

-87% 

-88% 

-89% 

-90% 

Discuss  Picture  with  Battle  Force  Units 

-98% 

-99% 

-99% 

-99% 

Relay  sub-total 

-97% 

-98% 

-98% 

-99% 

Totals 

212% 

240% 

273% 

404% 

Table  1.  Detailed  ROI  Estimates  for  AEGIS 
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AS  IS 

TO  BE 

RAD  1 

RAD  2 

CORRELATE 

Obtain  Link  Information 

3393% 

3280% 

3026% 

2598% 

Identify  "Same  Contact,  Multiple  Track" 

-91% 

-91% 

2530% 

2158% 

Verify  Other  Track  Sources 

-95% 

-95% 

-96% 

-96% 

Correlate  sub-total 

1174% 

1131% 

1512% 

1301% 

TRACK 

Monitor  Suspect  Tracks 

-98% 

-99% 

-99% 

-99% 

Update  Tracks 

-96% 

-97% 

546% 

475% 

Update  GCCS-M 

-98% 

14% 

10% 

1% 

Track  sub-total 

-98% 

-96% 

-53% 

-60% 

IDENTIFY 

Verify  IFF  signal 

790% 

757% 

692% 

595% 

Verify  EW  emissions 

-90% 

-91% 

-91% 

474% 

Verify  Point  of  Origin 

-98% 

3689% 

3405% 

2967% 

Match  Against  ATO 

-98% 

3813% 

3510% 

3049% 

Match  Against  CommAir  Profile 

926% 

896% 

816% 

688% 

Match  Against  Intel  Information 

-97% 

-97% 

-97% 

3688% 

Examine  Kinematic  Data 

-96% 

-96% 

-96% 

-96% 

Obtain  Visual  ID 

-100% 

-100% 

-100% 

-100% 

Conduct  Verbal  Query 

-99% 

-99% 

-99% 

-99% 

Identify  sub-total 

12% 

59% 

48% 

316% 

RELAY 

Send  Over  Links 

-82% 

-83% 

-84% 

-86% 

Discuss  Picture  with  Battle  Force  Units 

-99% 

-99% 

-99% 

-99% 

Relay  sub-total 

-97% 

-98% 

-98% 

-98% 

Totals 

212% 

237% 

271% 

399% 

Table  2.  Detailed  ROI  Estimates  for  SSDS 


Table  3  discusses  the  potential  impact  of  an  OA  approach  on  AEGIS  and  SSDS. 


“As  Is” 

“To  Be” 

“Radical  1” 

“Radical  2” 

None 

Limited  re-engineering 

Significant  re-engineering 

Substantial  re-engineering 

Technology 

Impact 

None 

•  Info  provided  in  ATO  could  be 
upgraded  into  AEGIS, 
reducing  manpower 
requirements. 

•  Enables  greater  sensor  and 
data  integration,  providing 
enhanced  correlation  in 
pinpointing  origin  of  aircraft  or 
ship. 

•  Streamlined  system  automatically 
updates  tracks. 

•  Increased  information-sharing  and 
collaborative  technology  allows  for 
automatic  correction  of  multiple  tracks 
per  target. 

•  Continuously  updates  tracks,  allowing 
for  pinpoint  accuracy. 

•  Collaborative  technology  minimizes 
possibility  of  multiple  tracking  of 
targets. 

•  Includes  changes  from  “To  Be” 

•  Collaborative  technology  automatically 
updates  ship’s  systems  with  Intel 
information. 

•  Electronic  communication  of  data  from  EW 
to  CIC  personnel  facilitates  COTS-based 
environment  that  easily  upgrades  to 
accommodate  greater  processor  speeds. 

•  Greatly  enhances  CIC  efficiency  through 
more  timely  SA. 

•  Includes  changes  from  “To  Be”  and 
“Radical  1.” 

Potential 

Benefits* 

•  Reduces  maintenance  costs. 

•  Frees  watch-standers  to 
perform  other  tasks  while 
providing  faster  data  flow. 

•  Increases  accuracy  of  tracking  targets. 

•  Substantial  re-engineering  leads  to  drastic 
reduction  in  watch-stander  work  time, 
greatly  reducing  human  error  and  further 
decreasing  maintenance  costs. 

Table  3.  Potential  Benefits  of  OA  Combined  with  Reengineering  of  Track 

Management  Operations 
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•  strategy  C:  Leave  and  Layer  is  the  most  promising  strategy  with  lowest 
total  costs.  It  has  the  highest  potential  rate  of  return  with  a  valuation  of 
$58.8  billion  and  4.9  times  the  potential  return  than  Strategy  A.  Strategy  B 
has  a  valuation  of  $23.2  billion,  while  Strategy  A  has  the  lowest  valuation  at 
$12  billion. 

Table  4.  Real  Options  Valuation  Results:  Strategies  A-C 


Strategy  A 

strategy  B 

Strategy  C 

STRATEGIC  OPTION 

“As  Is” 

DDX  OA 

“Develop  and  Retrofit” 

AEGIS  OA 
“Leave  &  Layer” 

Net  Present  Value 

$12.00B 

$6.38B 

$27.52B 

Real  Options  Value 

$12.00B 

$23.16B 

$58.84B 

Total  Cost 

$10.00B 

$24.00B 

$9.09B 

Strategic  Real  Options- 
based  Relative  Return  Ratio 

1.00 

1.9 

4.9 

•  Collaboration  is  critical.  OA  as  an  acquisition,  development  and 
deployment  framework  will  not  succeed  without  the  support  of  a  collaborative 
infrastructure  to  facilitate  the  introduction  of  multiple  large,  medium  and 
smaller  players  and  their  necessary  interactions  with  users  of  the  systems 
(e.g.,  warfighters),  the  acquisition  community,  and  Navy  leadership. 
Significant  investments  will  be  required  for  the  infrastructure  necessary  to 
enable  all  parties  (acquirers,  users,  developers)  to  collaborate  easily  and 
effectively  in  an  OA  model. 

•  Performance  monitoring  is  required.  If  the  performance  of  acquisition 
strategies  is  not  monitored  over  time,  the  probability  of  success  will  be  greatly 
reduced.  Performance  measurement  systems  (i.e.  performance  accounting 
software),  in  conjunction  with  predictive  forecasting  software  programs, 
provide  additional  analytic  support  to  IWS  systems-acquisition  strategies. 

These  research  results,  along  with  components  of  the  KVA+RO  framework  and  key 
findings  from  the  analysis,  are  summarized  in  this  report.  ® 


Summary 

IWS  systems  developed  in  a  closed,  proprietary  model  have  performed  well  and 
provide  substantial  returns.  However,  a  new  paradigm  is  required  to  maintain  military 
superiority  and  wage  information-age  warfare.  Through  open  architected  system 
development  and  open-business  models,  benefits  such  as  reusable  code,  lower 
maintenance-upgrade  costs,  and  greater  vendor  flexibility  in  supporting  system  module 
upgrades  could  be  derived.  Moreover,  the  Navy  can  leverage  new  technology  by  quickly 
adopting  it  to  warfighter  needs. 

This  study  found  that  the  “leave  and  layer”  option  for  IWS  replacement  provided  the 
lowest  costs,  highest  ROIs,  and  highest  strategic  options  value  with  the  lowest  risk.  The 


The  accuracy  of  our  analysis  is  dependent  on  data  and  information  provided  by  subject-matter  experts.  KVA  analysis 
includes  tests  of  the  reliability  of  their  estimates. 
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results  recommend  use  of  the  OA — leave  and  layer  IWS  replacement  approach  to  support 
reengineered  warfighting  processes. 

Introduction 

Naval  Open  Architecture  (Naval  OA)  is  a  multi-faceted,  enterprise-wide  business 
model  and  product-line  strategy  designed  to  fully  capitalize  on  “open”  computer  design 
principles  and  architectures.  Expanding  on  the  technological  open  architecture  (OA)  model, 
Naval  OA  leverages,  “open  business  models  for  the  acquisition  and  spiral  development  of 
new  systems  that  enable  multiple  developers  to  collectively  and  competitively  participate  in 
cost-effective  and  innovative  capability  delivery  to  the  Naval  Enterprise”  (Mullen,  2006, 
August  28).  The  new  OA  business  model  requires  a  greater  degree  of  collaboration  among 
customers  (e.g.,  warfighter),  builders  (e.g.,  small,  medium,  and  large  technology 
companies),  and  buyers  (e.g.,  the  acquisition  community)  than  the  existing  closed, 
proprietary  IWS  business  model.  In  the  migration  to  an  OA  business  model,  billions  of 
dollars  in  hardware  and  software  development  expenditures,  along  with  subsequent 
maintenance  costs,  are  at  stake. 

To  understand  the  potential  impact  of  OA  technology  and  business  models  on  naval 
warfighting  processes  and  procedures,  analytical  tools  are  critical  for  decision-makers  as 
they  manage  their  portfolio  of  options.  Portfolio  management  requires  that  these  tools 
quantify  the  risks,  costs,  and  net  value  of  potential  IWS  acquisitions.  The  tools  must  be  able 
to  help  identify  where  gaps  exist  in  current  processes  and  to  project  anticipated  returns  on 
investments  to  fill  those  gaps. 

This  study  describes  research  conducted  at  the  Naval  Postgraduate  School  (NPS) 
using  the  Knowledge  Value  Added/Real  Options  (KVA+RO)  valuation/risk  portfolio 
management  framework.  KVA+RO  is  a  comprehensive  measurement  tool  set  that  defines, 
measures  and  evaluates  total  value  of  given  IWS  acquisitions.  It  captures  data  across  a 
spectrum  of  organizations  to  compare  outputs,  processes,  capabilities,  risks,  costs,  and 
value  (i.e.  comparable  revenue).  KVA+RO  analytically  quantifies  uncertainty  and  risk 
elements  inherent  in  predicting  the  future,  includes  ways  to  mitigate  these  risks  through 
strategic  options  and  by  analytically  developing  and  allocating  budgets  to  optimize  project 
portfolios.  Understanding  uncertainties  and  mitigating  the  potential  impact  of  risks 
significantly  improves  the  likelihood  of  successful  acquisition  decisions. 

In  this  study,  KVA+RO  is  used  to  assess  the  implications  of  OA  on  SA  procedures 
onboard  the  AEGIS  and  SSDS  platforms.  Focusing  on  systems  interoperability,  KVA 
methodology  is  first  applied  to  generate  knowledge-based,  ROI  metrics  on  individual  sub¬ 
processes  and  watch  stations  involved  in  track-management  processes.  The  potential 
impact  of  OA  on  track  management  processes  and  sub-processes  is  analyzed  under 
several  scenarios:  “To  Be,”  “Radical  1,”  and  “Radical  2”for  AEGIS  and  SSDS.  Potential 
investments  are  then  evaluated  for  AEGIS  through  real  options  analysis,  resulting  in  net 
present  value  (NPV)  of  three  strategic  alternatives  ranging  from  $6.4  billion  to  $27.5  billion 
over  a  nine-year  period  and  options  valuations  of  from  $23.2  to  $58.8  billion. 

Lessons  from  the  "Open”  Solutions  Movement 


Disruptive  forces  and  accelerating  shifts  in  technology  have  enabled  organizations  to 
leverage  open  technology  platforms  to  achieve  greater  productivity  and  efficiency  levels. 
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These  “open”  solutions  offer  new  possibilities  for  solving  business  problems,  provide 
business  interoperability  by  standardization  and  technology  transparency,  and  decrease 
time  to  market  for  key  products  and  services.  Organizations  are  adopting  open  technology 
platforms  and  open-source  software  for  critical  business  needs,  moving  into  mainstream 
business  practices  in  corporations  such  as  IBM,  Google,  Intel,  JPMorgan  Chase,  Merrill 
Lynch  and  Pfizer. 

One  manifestation  of  the  movement  toward  “openness,”  yet  to  be  embraced  by  the 
Department  of  Defense  (DoD),  is  the  open-source  software  movement.  Germany,  Australia, 
the  United  Kingdom,  Finland,  Norway,  Canada,  China,  Japan  and  Brazil  are  among  the 
increasing  number  of  governments  embracing  open-source  software.  Open-source 
software  is  growing  at  such  a  rate  that  it  represents  the  most  significant  all-encompassing 
and  long-term  trend  that  the  software  industry  has  seen  since  the  early  1980s,  according  to 
a  recent  study  by  International  Data  Corporation  (IDC).  IDC’s  survey  of  over  5,000 
developers  in  1 16  countries  found  that  open  source  software  is  being  used  by  71%  of 
developers  in  the  world  and  is  in  production  at  54%  of  their  organizations. 

Open-source  software  development  site  SourceForge.net  reported  more  than 
129,000  projects  in  2006,  up  from  1 ,362  projects  in  2000.  Google’s  2006  “Summer  of  Code” 
open-source  initiative  has  630  collaborative  projects  pumping  more  than  $3  million  back  into 
the  open-source  community.  There  are  now  more  than  55  Open-Source  Initiative  (OSI) 
certified  open-source  licenses  available  given  the  popularity  of  open-source  software. 

The  success  of  this  movement  has  been  predicated  on  the  ability  of  the  multiple  parties 
involved  to  easily  collaborate  across  organizational,  field-specific,  and  national  boundaries. 
While  this  approach  to  software  development  may  not  directly  apply  to  security-sensitive 
systems  such  as  IWS,  lessons  can  be  learned  by  examining  the  results  of  this  movement  in 
the  commercial  world. 


Collaboration  is  Key 

As  with  the  use  of  the  OA  technology  and  business  model,  open-source  is  built  on 
the  tenants  of  open  access  and  collaboration.  The  lessons  learned  when  “openness”  is 
applied  to  system  development  and  business  models  from  the  open-source  movement  is 
that  such  approaches  allow  access  to  a  wider  development  community  that  can  adapt, 
improve  and  fix  software  at  a  faster  and  more  agile  pace  than  can  a  proprietary  vendor. 
Organizations  are  also  not  locked-into  one  vendor  or  product. 

Google,  for  example,  has  acknowledged  that  the  open  architecture  of  Google  Maps, 
allowing  external  developers  to  build  applications  on  top  of  it,  greatly  contributed  to  the 
mapping  service’s  functionality  and  diversity  at  a  greater  level  than  the  company  could  have 
done  internally  (Perez,  2006,  March  6).  In  2005,  IBM  opened  access  to  500  corporate 
software  patents,  forfeiting  $10  million  dollars  in  annual  royalties.  According  to  IBM, 
technological  advances  are  often  dependent  on  shared  knowledge,  standards  and 
collaborative  innovation  (IBM,  2005,  January  1 1 ).  IBM  believes  that  by  being  allowed 
access  to  those  patents,  open-source  developers  will  help  foster  continued  innovation. 


According  to  OSI,  the  most  commonly  used  licenses  are:  Apache  License,  GNU  Generai  Pubiic  License  (GPL),  GNU 
Lesser  General  Public  License  (LGPL),  Modified  BSD  (Berkeley  Software  Distribution)  License  (new  BSD)  and  Mozilla 
Public  License  (MPL).  MPL  is  the  most  widely  used  since  1998.  A  NASA  license  is  also  available  (OSI,  2006). 
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It’s  critical  to  note  that  the  full  potential  of  OA  and  open-business  model  approaches 
cannot  be  achieved  without  a  basic  collaborative  technology  infrastructure.  The  ease  with 
which  all  parties  share  ideas,  compare  requirements,  develop  solutions,  test  system 
capabilities,  and  finally  participate  jointly  in  deploying  systems  is  dependent  on  their 
commitment  to  collaborate  openly  and  fully  with  each  other.  This  can  only  be  facilitated, 
realistically  and  practically,  through  collaborative  technology.  Benefits  from  this  type  of 
approach  have  been  previously  demonstrated  in  shipyard  planning  where  the  use  of  product 
lifecycle  management  (PLM)  collaborative  software  added  tremendous  potential  value  to  the 
process  (Komoroski,  C.,  Housel,  T.,  Horn,  S.,  &  Mun,  J.,  2006,  October). 


Open  Architecture  and  the  Department  of  Defense 

Computer  software  plays  a  critical  component  in  maintaining  the  nation’s  defenses. 
For  example,  less  than  10%  of  its  functionality  was  provided  by  software  when  the  F-4 
fighter  was  developed  in  the  1960’s;  at  least  80%  of  the  F/A-22’s  functionality  is  software 
related  (GAO,  2004,  March).  Although  the  DoD  spends  billions  of  dollars  to  develop  and 
maintain  rights  to  millions  of  lines  of  code,  such  software  cannot  be  accessed  or  modified  by 
anyone  but  the  original  vendor  because  of  its  proprietary  nature  (Payton,  2006,  August  14). 
Moreover,  the  DoD  will  spend  as  much  as  $12  billion  on  reworking  software  for  major 
weapons  acquisitions  programs — 30%  of  its  estimated  budget  of  $40  billion  for  research, 
development,  testing  and  evaluation  in  Fiscal  Year  2006  (Wait,  2006,  July  3). 

Consequences  resulting  from  the  lack  of  OA  and  open  business  models  include: 

■  increased  development  and  maintenance  costs  for  information  technology; 

■  lock-in  to  obsolete  proprietary  technologies; 

■  inability  to  extend  existing  capabilities  in  months  versus  years;  and 

■  lack  of  interoperability  due  to  opacity  and  stove-piping  of  information  systems.  (Flerz, 

J.C.,  Lucas,  M.,  &  Scott,  J.,  2006) 

The  DoD  has  at  least  115  open-source  software  applications  used  in  more  than  250 
applications.  However,  IWS  software  acquisitions  are  still  made  with  the  same  industrial- 
age  business  models  used  to  acquire  ships,  tanks  and  other  physical  machinery  (Payton, 
2006,  August  14).  The  traditional  business  model  of  purchasing  physical  goods  and 
services  falls  short  when  applied  to  acquiring  digital  assets  like  IWS  technology.  New 
business  models  are  required  to  acquire  IWS  technology  to  wage  information-age  warfare 
requiring  responsiveness  and  agility,  according  the  Deputy  Under  Secretary  of  Defense  for 
Advanced  Systems  and  Concepts,  Dr.  Sue  Payton  (Payton,  2006,  August  14).  Moving  to 
an  open  architecture  model  maximizes  IT  acquisitions  by  saving  development  dollars, 
reducing  development  cycles,  and  fostering  new  and  innovative  solutions  and  capabilities. 

Modern,  analytical  tools  are  also  necessary  to  deploy  an  open  solutions  business 
strategy.  These  tools  must  measure  the  operational  value  of  a  system  from  the  warfighter’s 
perspective,  identify  areas  of  deficiencies  in  capabilities,  and  flag  areas  for  process 
improvement.  Traditional  measurement  tools  used  for  cost  analysis  cannot  calculate  the 
total  value  of  a  system,  particularly  operational  value  provided  by  specific  process 
improvements.  At  the  tactical  level,  an  operator  does  not  define  capabilities  merely  in  cost 
terms  but  also  in  time,  efficiency  and  effectiveness  gains  like  processing  more  targets  within 
a  given  time  period.  Given  new  potential  threats,  such  as  “swarm”  attacks  where  there  may 
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be  thousands  of  targets  at  any  one  time,  the  warfighter’s  perspective  in  deveioping  open 
and  agiie  systems  is  critical.  Focusing  on  the  potentiai  cost  reductions  from  the  OA  and 
open  business  models  approach  may  lead  developers  and  acquirers  away  from  the  reai 
needs  of  the  warfighter.  Maintaining  a  focus  on  the  potentiai  vaiue  produced,  in  addition  to 
potentiai  cost  savings,  is  critical  to  the  success  of  OA  and  open  business  modeis. 

KVA+RO  Framework 

The  KVA+RO  methodology  provides  an  equal  focus  on  the  potential  value  and  cost  of  new 
IWS  systems.  KVA+RO  measures  operating  performance,  cost-effectiveness,  return  on 
investments,  risk,  real  options  (capturing  strategic  flexibility),  and  analytical  portfolio 
optimization.  In  this  study,  it  was  applied  to  the  problem  of  finding  the  most  promising 
solution  for  replacing  aging  IWS  systems  such  as  AEGIS  and  SSDS  to  support 
reengineered  warfighting  capabilities.  KVA+RO  analysis  empowers  decision-makers  and 
supports  IWS  acquisition  strategies  by  providing  performance-based  data  and  scenario 
analysis.  Analyses  like  ROI  on  individual  IWS  projects  and  programs,  as  well  as  processes 
and  sub-processes  (e.g.,  track  management  processes  in  the  present  study)  supported  by 
IWS  systems  can  be  examined  within  a  portfolio  of  acquisitions  framed  through  the 
KVA+RO  methodology.  An  overview  of  the  framework  is  shown  in  Figure  1 . 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  - 134  - 


Analyzes/Forecasts 
Risks  and  Projects 
Potential  Value 


Captures  Process 
Data 


Establishes 
Market  Comps 


Figure  1.  NFS  Valuation  Framework 

The  framework  has  been  used  in  a  variety  of  NPS  analyses,  including  evaluating  the 
potential  impact  of  commercial-off-the-shelf  (COTS)  technology  on  naval 
maintenance/modernization  processes.  In  the  study  involving  one  specific  area  of  shipyard 
planning  for  maintenance  alterations,  cost  savings  were  projected  to  exceed  $40  million  per 
year  and  manpower  requirements  drastically  reduced  with  commercial-off-the-shelf,  three- 
dimensional  scanning/visualization  technology  and  collaborative  PLM  technology 
(Komoroski,  C.,  Housel,  T.,  Horn,  S.,  &  Mun,  J.,  2006,  October).  Key  components  of  the 
NPS  Valuation  framework  are  further  discussed  in  this  section. 


Knowledge-based  Metrics:  Knowledge  Value  Added  (KVA) 

KVA  measures  the  value  provided  by  human  capital  assets  and  IT  assets  (e.g.,  IWS 
systems  +  human  operators)  by  an  organization,  process  or  function  at  the  sub-process 
level.  It  monetizes  the  outputs  of  all  assets,  including  intangible  knowledge  assets. 
Capturing  the  value  embedded  in  an  organization’s  core  processes,  employees  and  IT 
enables  the  actual  cost  and  revenue  of  a  product  or  service  to  be  calculated.  Figure  2 
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identifies  the  types  of  assets  used  to  produce  output;  outputs  can  be  products  or  services 
produced  by  that  organization. 


Figure  2.  Measuring  Output 


Figure  3  shows  how  KVA  process  costing  differs  from  traditional  accounting 
methods. 


I 


.g 

3 


Traditional  Accounting 

KVA  Process  Costing 

Compensation 

$5,000 

Review  Task 

$1,000 

Benefits/OT 

1,000 

Determine  Op 

1,000 

Supplies/Materials 

2,000 

Input  Search  Function 

2,500 

Rent/Leases 

1,000 

Search/Collection 

1,000 

Depreciation 

1,500 

Target  Data  Acq 

1,000 

Admin.  And  Other 

900 

Target  Data  Processing 

2,000 

Total 

$11,400 

Format  Report 

600 

Quality  Control  Report 

700 

Transmit  Report 

1,600 

Total 

$11,400 

I 

§ 


Figure  3.  Comparison  of  Traditional  Accounting  versus  Process-based  Costing 


As  seen  in  Table  1 ,  total  value  is  captured  in  two  key  metrics:  ROI  and  ROK.  While 
ROI  is  the  traditional  financial  ratio,  ROK  identifies  how  a  specific  process  converts  existing 
knowledge  into  process  outputs  so  decision-makers  can  quantify  costs  and  measure  value 
derived  from  investments  in  productive  assets.  A  higher  ROK  signifies  better  utilization  of 
knowledge  assets.  If  IT  investments,  such  as  existing  IWS  systems,  do  not  improve  the 
ROK  value  of  a  given  process,  steps  must  be  taken  to  improve  that  process’s  function  and 
performance. 
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Metric 

Description 

Type 

Caicuiation 

Return  on  Knowledge 
(ROK) 

Basic  productivity,  cash¬ 
flow  ratio 

Sub-corporate,  process-level 
performance  ratio 

Outputs-Benefits  in  Common  Units 

Cost  to  Produce  Output 

Return  on  Investment 
(ROI) 

Same  as  ROI  at  the  sub¬ 
corporate,  process  level 

Traditional  investment 
finance  ratio 

(Revenue-Investment  Cost) 

Investment  cost 

Table  1.  KVA  Metrics 


Risk  Analysis:  Real  Options  (RO) 

Potential  strategic  investments  can  then  be  evaluated  with  real  options  analysis 
based  on  KVA  data.  This  analysis  is  a  robust  and  analytical  process  incorporating  the  risk 
identification  (applying  various  sensitivity  techniques),  risk  quantification  (applying  risk- 
based  Monte  Carlo  simulation),  risk  valuation  (applying  real  options  analysis),  risk  mitigation 
(utilizing  real  options  framing),  and  risk  diversification  (employing  analytical  portfolio 
optimization)  using  the  Real  Options  SLS  and  Risk  Simulator  software  programs.  Figure  4 
reflects  the  complex  calculations  for  integrated  risk  analysis  in  KVA+RO. 
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coupled  with  Monte  Carlo  simulation  is 
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...real  options  analytics  are  calculated 
through  binomial  lattices  and  closed- form 
partial-differential  models  with 
simulation. . . 


. .  .the  relevant  projects  are 
chosen  for  real  options 
analysis  and  the  project  or 
portfolio  real  options  are 
framed. . . 


7.  Portfolio  optimization  and  asset 
allocation 


. .  .stochastic  optimization  is  the  next 
optional  step  if  multiple  projects  exist  that 
requires  efficient  asset  allocation  given 
some  budgetary  constraints. . .  useful  for 
strategic  portfolio  management. . . 


8.  Reports  presentation  and 
update  analysis 


Figure  4.  Integrated  Risk  Analysis 


Beyond  Concept:  KVA+RO  Implementations 

Moving  beyond  a  concept  stage,  the  KVA+RO  framework  is  being  implemented  in 
SPAWAR  and  in  the  Army  Rapid  Equipping  Force  project.  KVA+RO  is  being  used  in  both 
projects  to  improve  processes,  reduce  cycle-times  and  costs,  and  increase  value  by 
allowing  Navy  executives  to  acquire  intelligence  systems  via  a  portfolio  approach  and  by 
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getting  the  Army  troops  in  the  field  (i.e.,  Iraq  and  Afghanistan)  what  they  need  very  quickly 
through  new  rapid  acquisition  processes. 

Proof-of-Concept  Case  Study:  Situational  Awareness  Onboard 
AEGIS  and  SSDS  Platforms^ 

This  proof  of  concept  case  study  is  designed  to  assist  PEO  IWS,  Open  Architecture 
Division,  with  its  mandate  of  implementing  OA  in  the  Navy.  The  case  is  prepared  from  a 
warfighter  perspective  because  the  value  of  OA  must  be  proven  to  the  ultimate  end-user. 
This  perspective  also  permits  a  review  of  how  OA  can  lead  to  flexible  system  acquisition  and 
deveiopment  to  enable  reengineered  processes  that  wiii  provide  better  performance  in  core 
warfighting  processes  such  as  SA. 

In  a  multi-phased  approach,  KVA+RO  was  applied  to  SA-track  management 
procedures  used  in  the  AEGIS  and  SSDS  platforms.  As  illustrated  in  Figure  5,  the  totai 
functionaiities  of  AEGIS  and  SSDS  systems  are  very  broad  so  we  focused  our  research  on 
SA  because  it  appeared  to  be  the  most  promising  area  for  upgrading  and  reengineering 
according  to  subject  matter  experts.  The  goals  of  this  research  were  to: 

■  Demonstrate  the  efficacy  of  the  KVA-RO  framework  to  evaluate  reengineering 
designs  for  warfighting  core  processes  (i.e.,  SA-track  management)  in  terms  of  the 
ROI  and  strategic  option  value  of  various  OA  approaches  to  replacing  aging  IWS 
systems. 

■  Determine  which  elements  of  the  track  management  process  could  be  reengineered 
using  an  OA  approach 

■  Identify  areas  of  improvement  for  current  surface  ship  track-management 
processes  using  the  existing  two  IWS  systems:  AEGIS  and  SSDS. 


Information  collected  from  subject-matter  experts  (SMEs)  from  Surface  Warfare  Fleet  and  training  commands  at  Dahlgren 
(AEGIS)  and  Wallops  Island  (SSDS).  Information  gathered  from  SMEs  then  aggregated  to  provide  an  average  for  each 
process  to  ensure  accuracy.  Additional  information  collected,  including  process  flow  diagrams,  use-case  diagrams  and 
literature  review,  to  develop  baseline  data. 
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Figure  5.  Planned  Reuse  of  Aegis  and  SSDS  in  DD(X) 

(MacRitchie,  2005,  June  29,  p.  12) 


Background 

In  the  late  1990s,  the  DoD  articulated  a  vision  for  network-centric  (or  “net-centric”) 
warfare  in  which  networking  military  forces  wouid  facilitate  information  sharing  and 
coilaboration,  ieading  to  enhanced  SA.  Information  superiority  is  vital  to  enhanced  SA,  rapid 
decision-making,  improved  efficiency,  speedy  execution  and  mission  effectiveness.  A  high 
degree  of  interoperabiiity  is  required  to  achieve  information  superiority.^^  Lack  of 
interoperabiiity  between  the  services  makes  it  difficuit  for  the  warfighter  to  distinguish  “friend” 
from  “foe”  and  to  make  critical  decisions — potentiaiiy  deiaying  miiitary  response  times  or 
contributing  to  lethai  mistakes.  Figure  6  shows  a  scenario  in  which  a  sea-based  system  and 
a  iand-based  system  are  tracking  aircraft  and  are  unabie  to  integrate  their  views  of  a 
battlefield. 


The  DoD  defines  interoperability  as  the  abiiity  of  systems,  units,  or  forces  to  exchange  data,  information,  materiel,  and 
services  to  enable  them  to  operate  effectively  together. 
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Figure  6.  Scenario  for  Tracking  Threats  without  Benefit  of  Interoperable 

Systems 

(GAO,  2006,  January,  p.  6) 


To  achieve  information  superiority  and  enable  net-centric  warfare,  the  DoD  has  been 
developing  the  Global  Information  Grid  (GIG).  Interoperable  systems  are  critical  to  the  GIG 
for  joint  military  operations  and  to  allow  users  access  to  data  on  demand,  to  share 
information  in  real-time  and  to  collaborate  in  decision-making  from  almost  any  location. 
Development  of  this  capability  is  ultimately  dependent  on  support  from  reengineered  SA 
systems  within  IWS  suites. 


Naval  Challenges 

The  Navy  must  develop  architectures  that  meet  the  integration  requirements  for  the 
GIG.  This  is  a  critical  requirement  in  designing  and  implementing  new  IWS  systems  built 
within  the  OA  framework.  In  addition,  the  Navy  must  resolve  the  following  issues  that  are  a 
result  of  legacy  technology  systems,  such  as  the  aging  IWS  systems. 


■  Limited  computational  and  operational  capability.  Systems  operating  at  99% 
capacity  in  non-stressed  environments. 

■  Difficulty  or  inability  to  add  new  warfighting  missions.  “Stove-piped”  systems 
diminish  interoperability  and  ability  to  meet  national  security  threats. 
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■  Prohibitive  software  maintenance  costs.  Some  $3+  billion  spent  across  Future 

Years  Defense  Plan  in  PEO  IWS  to  develop  and  maintain  computer  programs. 

Additional  testing  and  certification  required  when  new  capabilities  added. 

The  Navy  has  historically  acquired  IWS  systems  that  are  proprietary  in  design  and 
engineering,  require  unique  parts,  equipment,  and  services  to  support  them,  are  supported 
by  a  limited  number  of  suppliers,  and  become  very  expensive  to  maintain  (Strei,  2003,  April 
1).  Moreover,  systems  and/or  platforms  were  entirely  eliminated  rather  than  upgraded  or 
modernized  because  of  prohibitive  costs.  Rapid  technological  obsolescence,  compounded 
by  exorbitantly  escalating  costs  for  proprietary  systems  are  daunting  challenges  because 
design,  development,  and  acquisition  timelines  can  span  as  much  as  15  years  before  a 
military  platform  reaches  operating  forces  (Strei,  2003,  April  1). 


Naval  Open  Architecture  and  Open  Business  Modeis 

OA  and  open-business  models  propel  the  Navy  into  the  next  era  of  joint 
interoperability  while  resolving  legacy  issues  that  provide  new  benefits,  including: 

■  Lower  lifecycle  costs  for  IWS  systems.  Total  cost  of  ownership  decreases  due  to 
increased  maintainability,  interoperability,  upgradeability  and  use  of  a  wider  variety  of 
vendors. 

■  Better  performing  systems.  Ability  to  rapidly  upgrade  hardware  and  software  with 
the  latest  technology  enables  greater  capabilities,  efficiencies  and  interoperability  to 
enable  reengineered  warfighting  processes. 

■  Improved  interoperability  for  joint  warfighting.  Software  reuse  and  modularity 
facilitates  interoperability  between  systems  that  use  an  open  architecture  framework. 

■  Facilitating  competition  and  increasing  cooperation  between  commercial  and 
military  electronics  industries.  Moving  away  from  proprietary  systems  enables  a 
broader  range  of  ideas  and  technological  solutions. 

Guiding  principles  behind  Naval  OA  are  modularity,  reusability,  interoperability, 
lifecycle  affordability  and  collaboration  and  competition.  In  adopting  an  open,  OA  strategy 
based  on  commercially  available,  non-proprietary  information  technology  (IT)  standards, 
interfaces  and  formats,  the  Navy  will  need  to  increase  collaboration  (e.g.,  supported  by 
readily  available  collaborative  product  lifecycle  management  technology)  to  spur  competition 
and  fuel  innovation  in  the  acquisition  lifecycle. 

Ease  of  collaboration  is  critical  to  Naval  OA  to  ensure  that  multiple  vendors  compete, 
including  the  smaller,  more  nimble  companies.  Collaboration  also  provides  the 
infrastructure  necessary  to  facilitate  all  parties  sharing  critical  requirements  and  performance 
information  to  reduce  system  modification,  re-fresh  or  replacement  cycle-times.  As  such, 
collaborative  capabilities  will  facilitate  moving  OA  beyond  a  purely  technical  focus  to  a  more 
encompassing  open-business  model,  one  advocated  by  CNO  Admiral  Mullen.  As  noted 
earlier.  Admiral  Mullen’s  vision  for  open  architecture  isn’t  limited  to  systems  built  to  a  set  of 
open  standards,  but  focuses  on  open-business  models  tapping  into  a  multiple-developer 
network  to  deliver  innovative,  cost-effective  and  rapid,  spiral  acquisition  capabilities  to  the 
Navy. 
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Migrating  to  an  OA  environment  has  been  slow,  however,  despite  the  Navy’s  early 
adoption  of  open  source  strategies.  Encouraged  by  the  cost-effective  advantages  gained 
through  the  Acoustic  Rapid  commercial  off-the-shelf  Insertion/Advanced  Processor  Build 
(ARCI/APB)  program.  Admiral  Mullen  noted  in  a  recent  memo  his  disappointment  with  the 
slow  pace  of  adoption  and  advocated  rapid  transition  to  the  open-business  model  (Fein, 
2006,  September  11). 


SA:  Track-management  Processes 

Track  management,  a  fundamental  capability  inherent  to  all  IWS  SA  capabilities  for 
surface  ships,  is  the  process  by  which  friendly  and  enemy  forces  are  detected,  identified, 
monitored,  updated  and  communicated  throughout  the  area  of  operations  (AOR).  The  track 
management  process  within  a  Combat  Information  Center  (CIC)  is  very  complex, 
sophisticated  and  involves  multiple  watch  stations  and  technological  systems.  AEGIS  and 
SSDS  have  different  SA  procedures  and  policies,  and  track-management  functions  within 
the  CIC.  Although  variations  exist  in  track-management  processes,  watch  stations  are  fairly 
consistent  on  both  AEGIS  and  SSDS  ships.  Figure  7  is  a  generalized  organizational  chart 
of  CIC  personnel  directly  involved  in  track-management  processes. 


Figure  7.  CIC  Organizational  Chart 


Track-management  processes  entail  various  sub-processes,  as  seen  in  Figure  8.^"^ 
This  graphic  is  an  aggregated  view  for  both  AEGIS  and  SSDS  platforms  consisting  of  four 
principal  processes  and  17  sub-processes. 


Although  watch  stations  talk  to  specific  tasks  and  responsibiiities,  in  an  actuai  CIC,  all  personnel  listed  can  be  actively 
involved  in  any,  or  all,  aspects  of  track  management  (correlation,  identification,  tracking,  and  relaying). 

Figure  derived  from  numerous  SMEs  from  AEGIS  and  SSDS  communities.  While  sub-processes  may  differ  from  ship  to 
ship,  SMEs  concluded  that  the  four  primary  processes  reflect  track  management  procedures  conducted  within  a  CIC. 
SMEs  agreed  that  the  17  sub-processes  shown  reflected  individual  tasks  appropriate  for  this  limited  research.  SMEs  also 
concluded  that  there  is  no  definitive  sequential  order  in  which  specific  tasks  occur;  however,  the  figure  provides  a  potential 
sequence. 
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Figure  8.  Track  Management  Sub-processes 
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Case  Study  Results:  Potential  Impact  of  OA 


The  potential  impact  of  OA  on  AEGIS  and  SSDS  platforms  was  calculated  through 
KVA+RO  in  a  muiti-phased  approach.  In  addition,  KVA  data  estimates  and  reai  options 
models  were  based  on  the  assumption  that  functionai  upgrades  would  primarily  be  in  the  SA 
area  with  the  remaining  modules  providing  at  least  the  same  capabilities  as  the  current 
AEGIS  IWS  system. 

KVA  methodology  was  first  applied  to  derive  potential  benefits  in  SA  processes 
within  AEGIS  and  SSDS-class  ships  (i.e.,  84  Destroyers  and  Cruisers).  Track-management 
sub-processes  by  process  category  (and  by  watch  station  in  Appendix  B)  were  evaluated 
under  four  improvement  scenarios  {“As  Is,”  “To  Be,”  “Radical  1,” and  “Radical  2”).  The 
following  assumptions  were  used  to  calculate  the  data: 

■  Integration  with  middleware  until  Category  3  Open  Architecture  Computing 
Environment  (OACE)  level  has  been  reached  for  systems  being  evaluated 

■  Use  of  OA  approach  to  developing  the  IWS  systems  and  use  of  Commercial-off-the- 
shelf  (COTS)  equipment 

Steps  in  calculating  KVA  data  were: 

1 .  Identify  core  processes  and  sub-processes. 

2.  Establish  common  units  and  level  of  complexity  to  measure  learning  time. 

3.  Calculate  learning  time  (i.e.,  knowledge  surrogate)  to  execute  each  sub-process. 

4.  Designate  sampling  time  period  long  enough  to  capture  representative  sample  of  the 
core  processes’  final  product  or  services  output. 

5.  Multiply  learning  time  for  each  sub-process  by  number  of  times  sub-process 
executes  during  sample  period. 

6.  Calculate  cost  to  execute  knowledge  (learning  time  and  process  instructions)  to 
determine  process  costs. 

7.  Calculate  ROK  (ROK=  Revenue/Cost)  and  ROI  (ROK=  Revenue-Cost/Cost). 

During  Phase  2,  real  options  analysis  focused  on  the  options  for  improving  {Leave 
and  Layer  option)  or  replacing  {Retrofitting  option)  the  AEGIS  IWS  system.  The  option  to 
continue  with  the  current  proprietary  systems  approach  was  provided  as  a  baseline  for 
comparison  purposes.  Future  research  would  allow  us  to  examine  all  AEGIS  modules  for 
potential  upgrading  and  would  likely  result  in  even  higher  ROI  estimates  as  well  as  real 
options  valuations. 
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KVA  Results:  ROK  and  ROI 


ROK  and  ROI  values  provide  insights  into  sub-processes  that  could  be  reengineered 
to  achieve  maximum  operational  efficiency.  Aggregated  results  for  AEGIS  and  SSDS  are 
shown  Figures  9  and  10.  The  “As  Is”  provides  a  baseline  ROI/ROK  performance  measure 
for  comparison  of  the  three  process  reengineering  designs.  The  three  redesigns  essentially 
represent  the  effects  of  increasing  levels  of  automation  in  the  track  management  process. 


AEGIS 


SSDS 


Figure  10.  KVA  Results:  ROK  and  ROI  Estimates 

OA  has  the  potential  to  provide  these  operational  performance  improvement 
benefits:  decreased  training  time  for  operators  of  systems,  decreased  “touch  time”  on 
processes  by  replacing  manual  processes  with  new  automated  capabilities,  and  increased 
efficiency  through  seamless  integration  of  multiple  system  components.  As  shown  in  Table 
2  below,  the  cumulative  impact  of  OA  on  the  track-management  processes  results  in 
significant  improvement  in  three  of  four  areas. 
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“To  Be” 

“Radical  1” 

“Radical  2” 

Cumulative 

Impact 

CORRELATE 

X 

X 

X 

TRACK 

X 

X 

X 

X 

IDENTIFY 

X 

X 

X 

X 

RELAY 

Table  2.  Process  Reengineering  Impacts:  Process  Level  and  Cumulative 

Tables  3  through  6  are  ROK  and  ROI  results  by  core  processes  and  sub-processes 
for  AEGIS  and  SSDS.  The  ROI  estimates  demonstrate  that  as  various  system  functionalities 
of  the  existing  track-management  process  are  upgraded,  a  corresponding  performance 
improvement  is  derived  in  those  areas.  This  more  detailed  analysis  suggests  where  OA 
based  system  upgrades  should  be  applied  to  achieve  the  best  results.  For  example,  the 
core  process  of  correlate  and  identify  have  the  greatest  potential  to  benefit  from  an  OA 
approach  to  system  development.  These  estimates  also  provide  the  basis  for  the  real 
options  analysis  projections  staged  over  a  9-year  period. 


AS  IS 

TO  BE 

RAD  1 

RAD  2 

CORRELATE 

Obtain  Link  Information 

3521% 

3407% 

3163% 

2733% 

Identify  "Same  Contact,  Multiple  Track" 

9% 

9% 

2161% 

1856% 

Verify  Other  Track  Sources 

5% 

5% 

5% 

4% 

Correlate  sub-total 

1284% 

1241% 

1606% 

1396% 

TRACK 

Monitor  Suspect  Tracks 

2% 

1% 

1% 

1% 

Update  Tracks 

3% 

3% 

461% 

410% 

Update  GCCS-M 

3% 

191% 

184% 

169% 

Track  sub-total 

2% 

6% 

42% 

36% 

IDENTIFY 

Verify  IFF  signal 

902% 

869% 

806% 

707% 

Verify  EW  emissions 

9% 

9% 

8% 

609% 

Verify  Point  of  Origin 

2% 

4221% 

3921% 

3432% 

Match  Against  ATO 

2% 

4306% 

3990% 

3482% 

Match  Against  CommAir  Profile 

963% 

935% 

863% 

743% 

Match  Against  Intel  Information 

3% 

3% 

3% 

3914% 

Examine  Kinematic  Data 

4% 

4% 

3% 

3% 

Obtain  Visual  ID 

0% 

0% 

0% 

0% 

Conduct  Verbal  Query 

1% 

1% 

1% 

1% 

Identify  sub-total 

108% 

160% 

150% 

426% 

RELAY 

Send  Over  Links 

13% 

12% 

11% 

10% 

Discuss  Picture  with  Battle  Force  Units 

2% 

1% 

1% 

1% 

Relay  sub-total 

3% 

2% 

2% 

1% 

TOTALS 

312% 

340% 

373% 

504% 

Table  3.  Detailed  ROK  Estimates  for  AEGIS 
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AS  IS 

TO  BE 

RAD  1 

RAD  2 

CORRELATE 

Obtain  Link  Information 

3493% 

3380% 

3126% 

2698% 

Identify  "Same  Contact,  Multiple  Track" 

9% 

9% 

2630% 

2258% 

Verify  Other  Track  Sources 

5% 

5% 

4% 

4% 

Correlate  sub-total 

1274% 

1231% 

1612% 

1401% 

TRACK 

Monitor  Suspect  Tracks 

2% 

1% 

1% 

1% 

Update  Tracks 

4% 

3% 

646% 

575% 

Update  GCCS-M 

2% 

114% 

110% 

101% 

Track  sub-total 

2% 

4% 

47% 

40% 

IDENTIFY 

Verify  IFF  signal 

890% 

857% 

792% 

695% 

Verify  EW  emissions 

10% 

9% 

9% 

574% 

Verify  Point  of  Origin 

2% 

3789% 

3505% 

3067% 

Match  Against  ATO 

2% 

3913% 

3610% 

3149% 

Match  Against  CommAir  Profile 

1026% 

996% 

916% 

788% 

Match  Against  Intel  Information 

3% 

3% 

3% 

3788% 

Examine  Kinematic  Data 

4% 

4% 

4% 

4% 

Obtain  Visual  ID 

0% 

0% 

0% 

0% 

Conduct  Verbal  Query 

1% 

1% 

1% 

1% 

Identify  sub-total 

112% 

159% 

148% 

416% 

RELAY 

Send  Over  Links 

18% 

17% 

16% 

14% 

Discuss  Picture  with  Battle  Force  Units 

1% 

1% 

1% 

1% 

Relay  sub-total 

3% 

2% 

2% 

2% 

Totals 

312% 

337% 

371% 

499% 

Table  4.  Detailed  ROK  Estimates  for  SSDS 
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AS  IS 

TO  BE 

RAD  1 

RAD  2 

CORRELATE 

Obtain  Link  Information 

3421% 

3307% 

3063% 

2633% 

Identify  "Same  Contact,  Multiple  Track" 

-91% 

-91% 

2061% 

1756% 

Verify  Other  Track  Sources 

-95% 

-95% 

-95% 

-96% 

Correlate  sub-total 

1184% 

1141% 

1506% 

1296% 

TRACK 

Monitor  Suspect  Tracks 

-98% 

-99% 

-99% 

-99% 

Update  Tracks 

-97% 

-97% 

361% 

310% 

Update  GCCS-M 

-97% 

91% 

84% 

69% 

Track  sub-total 

-98% 

-94% 

-58% 

-64% 

IDENTIFY 

Verify  IFF  signal 

802% 

769% 

706% 

607% 

Verify  EW  emissions 

-91% 

-91% 

-92% 

509% 

Verify  Point  of  Origin 

-98% 

4121% 

3821% 

3332% 

Match  Against  ATO 

-98% 

4206% 

3890% 

3382% 

Match  Against  CommAir  Profile 

863% 

835% 

763% 

643% 

Match  Against  Intel  Information 

-97% 

-97% 

-97% 

3814% 

Examine  Kinematic  Data 

-96% 

-96% 

-97% 

-97% 

Obtain  Visual  ID 

-100% 

-100% 

-100% 

-100% 

Conduct  Verbal  Query 

-99% 

-99% 

-99% 

-99% 

Identify  sub-total 

8% 

60% 

50% 

326% 

RELAY 

Send  Over  Links 

-87% 

-88% 

-89% 

-90% 

Discuss  Picture  with  Battle  Force  Units 

-98% 

-99% 

-99% 

-99% 

Relay  sub-total 

-97% 

-98% 

-98% 

-99% 

Totals 

212% 

240% 

273% 

404% 

Table  5.  Detailed  ROI  Estimates  for  AEGIS 


AS  IS 

TO  BE 

RAD  1 

RAD  2 

CORRELATE 

Obtain  Link  Information 

3393% 

3280% 

3026% 

2598% 

Identify  "Same  Contact,  Multiple  Track" 

-91% 

-91% 

2530% 

2158% 

Verify  Other  Track  Sources 

-95% 

-95% 

-96% 

-96% 

Correlate  sub-total 

1174% 

1131% 

1512% 

1301% 

TRACK 

Monitor  Suspect  Tracks 

-98% 

-99% 

-99% 

-99% 

Update  Tracks 

-96% 

-97% 

546% 

475% 

Update  GCCS-M 

-98% 

14% 

10% 

1% 

Track  sub-total 

-98% 

-96% 

-53% 

-60% 

IDENTIFY 

Verify  IFF  signal 

790% 

757% 

692% 

595% 

Verify  EW  emissions 

-90% 

-91% 

-91% 

474% 

Verify  Point  of  Origin 

-98% 

3689% 

3405% 

2967% 

Match  Against  ATO 

-98% 

3813% 

3510% 

3049% 

Match  Against  CommAir  Profile 

926% 

896% 

816% 

688% 

Match  Against  Intel  Information 

-97% 

-97% 

-97% 

3688% 

Examine  Kinematic  Data 

-96% 

-96% 

-96% 

-96% 

Obtain  Visual  ID 

-100% 

-100% 

-100% 

-100% 

Conduct  Verbal  Query 

-99% 

-99% 

-99% 

-99% 

Identify  sub-total 

12% 

59% 

48% 

316% 

RELAY 

Send  Over  Links 

-82% 

-83% 

-84% 

-86% 

Discuss  Picture  with  Battle  Force  Units 

-99% 

-99% 

-99% 

-99% 

Relay  sub-total 

-97% 

-98% 

-98% 

-98% 

Totals 

212% 

237% 

271% 

399% 

Table  6.  Detailed  ROI  Estimates  for  SSDS 
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Table  7  summarizes  how  OA  could  specifically  impact  the  sub-processes  deriving 
most  significant  improvements:  “Identify ‘Same  Contact,  Multiple  Track',”  “Update  GCCS- 
M,”  “Update  Track,”  “Verify  Point  of  Origin,”  “Match  against  ATO,”  “Verify  EW  Emissions” 
and  “Match  Against  Intel  Information.”  Each  watch  station  at  the  CIC  was  affected  (see 
Appendix  B). 


PROCESS 

SUB-PROCESS 

COMMENTS  POTENTIAL  IMPACT  OF  OA 

CORRELATE 

‘Identify  ‘Same 
Contact,  Multiple 
Track’ 

•  Reduces  reliance  on  manual  identification  of  multiple  tracks  and  updating  current  tracks. 

•  Automatically  corrects  anomaly  of  multiple  tracks  per  target  and  update  tracks.  Only  brief 
confirmation  by  the  watch  station  operator  necessary. 

TRACK 

“Update  GCCS-M” 

•  Enhances  operational  value  of  systems  through  reduced  time,  manpower  and  training 
required  to  conduct  process. 

TRACK 

“Update  Track” 

•  Reduces  reliance  on  manual  identification  of  multiple  tracks  and  updating  current  tracks. 

•  Automatically  corrects  multiple  tracks  per  target  anomaly  and  update  tracks,  resulting  in 
brief  confirmation  by  watch  station  operator. 

IDENTIFY 

“Verify  Point  of 
Origin” 

•  Enables  greater  sensor  and  data  integration,  providing  enhanced  correlation  in 
pinpointing  origin  of  aircraft  or  ship. 

•  Queries  point  of  origin  for  friendly  force  contacts  from  an  open  GCCS-M  system,  and 
interrogates  ATO  neutral-force  contacts  from  host  nation  airports  (assuming  data  format 
standardized  and  provided  by  host  nations.). 

•  Facilitates  interfaces  to  other  systems  to  provide  automated  query  for  point  of  origin. 

•  Frees  watch  standers  to  perform  other  tasks  while  providing  faster  data  flow. 

IDENTIFY 

“Match  Against 
ATO” 

•  Integrates  info  provided  in  ATO  into  the  AEGIS  and  SSDS  platforms,  greatly  reducing 
manpower  requirements. 

IDENTIFY 

“Verify  EW 
Emissions” 

•  Facilitates  COTS-based  environment  for  easier  upgrades  to  accommodate  greater 
processor  speeds. 

•  Enhances  CIC  efficiency  through  more  timely  SA. 

•  Frees  operators  to  perform  other  tasks. 

IDENTIFY 

“Match  Against  intei 
information” 

•  Streamlines  sub-process  with  automatic  updates  requiring  merely  manual  confirmation. 

Table  7.  Potential  Impact  of  OA  at  Sub-process  Levels 

Value-risk  Analysis:  Strategic  Rea!  Options  Analysis 

Real  options  analysis  was  performed  to  determine  the  prospective  value  of 
alternative  COAs  for  upgrading  the  AEGIS  IWS  in  track  management  over  a  nine-year 
period  with  KVA  data  inputs.  In  all  new  options  for  IWS  deployment,  it  was  assumed  that  a 
collaborative  technology  infrastructure  was  present  to  facilitate  the  use  of  the  OA  system 
and  open-business  model  approaches.^® 

Figure  1 1  illustrates  the  three  main  strategies  laid  out  as  a  real  options  map. 
Strategy  A  is  do  nothing,  leaving  everything  “As  Is”  with  the  ability  to  retire  ships  and  their 
AEGIS  systems  within  10  years.  Strategy  B  is  the  “DDX  Open  Architecture” (retrofit)  option 
with  new  development  within  the  first  three  years  at  a  cost  of  $8  billion.  Under  this  strategic 
path,  follow-up  is  required — with  retrofitting  costing  an  additional  $16  billion  within  6  years 


Estimates  are  based  on  historical  data  and  additional  information  provided  by  SMEs.  We  have  attempted  to  be  as 
conservative  as  possible  and  have  assumed  very  high  potential  volatility  in  both  of  the  new  IWS  development  options. 
Access  to  more  precise  performance  data  will  help  resolve  uncertainties  and  risks  over  time. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  - 149  - 


after  first-phased  new  development  has  been  completed.  Strategy  C  looks  at  the  “Leave 
and  Layer”  option  with  a  three-phased  sequentiai  compound  option  of  “To  Be,”  “  Radicai 
1,”,  and  “Radicai  2”  impiementation  within  9  years. 


Start 


Strategy  A 


AS-IS  strategy  where  the  older 
ships  are  retired  within  10 
years  and  no  further 
development  aetion  is  taken. 

Option  Value  =  $12B 
Total  Cost  =  $10B 


Retire  Ships 


Within  10  years 


Exit 


Do  nothing  for  now 


New 

Development 


- ►  Strategy  B  — 

DDX  Open  Arehiteeture 
approaeh  of  new  development 
within  3  years  followed  by 
retrofitting  within  6  years,  with 
an  option  to  exit  and  abandon 
before  the  retrofitting  phase. 
Option  Value  =  $23 .2B 

Total  Cost  =  $24B 


Inv  =  $8B 
Within  3  years 


Retrofitting 


Inv  =  $16B 

Anytime  within  6  years 


Exit 


Exit 


Stop  after  new  development 
without  retrofitting 


Do  nothing 


Leave  & 
Layer 


Strategy  C 

- ► 

Exit 

TO-BE 


Inv  =  $3.44B 
Anytime  within  3  years 


RADICAL 


Inv  =  $2.55B 
Anytime  within  6 years 
(after  To-Be) 


RADICAL  II 


Inv  =  $3.  IB 
Anytime  within  9  years 
(after  Radical) 


Exit 


Exit 


Do  not  eontinue 
development 


a  Leave  &  Layer  approaeh, 
followed  by  a  9-year  sequential 
option  of  stage-gate  investment 
of  To-Be,  Radieal,  and  Radieal 
II,  with  the  options  to  exit  and 
abandon  at  any  phase. 

Option  Value  =  $58.8B 
Total  Cost  =  $9.09B 


Do  not  eontinue 
development 


Exit 

Do  not  eontinue 
development 


Do  not  eontinue 
development 


Figure  11.  Strategic  Real  Options  Map 

Strategy  C  is  the  option  with  the  greatest  potential  value.  As  seen  in  Table  8  below, 
Strategy  C  provides  4.9  times  the  risk  adjusted  return  of  the  “As  Is”  strategy  versus  Strategy 
B  at  1 .9  times  the  return.  Strategy  C’s  incremental  approach  offers  the  lowest  risk  and 
numerous  benefits,  including  less  disruption  to  the  rest  of  system  and  deriving  benefits 
faster.  It  is  the  lowest  total  cost  alternative  with  costs  spread  over  a  nine-year  period,  yet 
the  program  reaps  incremental  benefits  from  various  functionality  improvements  throughout 
that  time.  This  strategy  provides  the  highest  NPV. 


Strategy  A 

Strategy  B 

Strategy  C 

STRATEGIC  OPTION 

“As  Is” 

DDX  OA 

“Develop  and  Retrofit” 

AEGIS  OA 
“Leave  &  Layer” 

Net  Present  Value 

$12B 

$6.38B 

$27.52B 
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Volatility 

0% 

80.5% 

86.3% 

Real  Options  Value 

$12B 

$23.155B 

$58.84B 

Strategic  Real  Options- 
based  Return  on  Investment 

N/A 

72.36% 

224.75% 

Total  Cost 

$10B 

$24B 

$9.09B 

Strategic  Real  Options- 
based  Relative  Return  Ratio 

1.0 

1.9 

4.9 

Notes: 


(1)  The  volatility  measure  quantifies  uncertainty  and  risk  levels  in  the  strategy  and  calibrates  them  to  account 
for  the  time  required  to  complete  the  entire  strategy. 

(2)  Strategic  real  options  values  are  also  computed,  accounting  for  value  of  open  architecture  as  laid  out  in 
the  phased-gate  development  process. 

(3)  Strategic  real  options-based  return  on  investment  looks  at  real  options  valuation  results  and  computes  ROI 
based  on  the  option  values  and  implementation  of  costs  each  phase.  The  higher  this  ROI  value,  the  more 
strategic  and  valuable  or  profitable  a  project. 

Table  8.  Real  Options  Valuation  Results:  Strategies  A-C 

“Leave  and  Layer”  allows  organizations  to  benefit  from  incremental  adoption.  Rather 
that  executing  a  plan  that  requires  everything  to  be  accomplished  at  once,  “leave  and  layer” 
enables  existing  systems  to  be  reused  successfully.  NAVFAC  successfully  adopted  the 
approach  to  provide  a  more  efficient,  lower  cost  contract  management  solution.  NAVFAC 
architected  a  technology  platform  to  allow  it  to  build  a  layer  of  Web-based  collaborative 
project  management  tools,  while  leveraging  existing  financial,  HR,  and  scheduling  systems. 
A  number  of  applications  were  developed,  including  the  collaborative  eProjects  application 
with  a  budget  of  $350,000  and  completion  in  10  months  (Oracle,  2004).  eProjects  provides 
one-click  schedule  and  cost  status.  Another  application,  eContracts,  automates  nearly  200 
redundant  screens  per  contract  action. 

In  our  analysis.  Strategy  B  has  the  highest  cost  due  to  high  up-front  costs  required  to 
build  the  system  within  the  first  five  years  without  deriving  any  benefits  from  the  new  system 
during  that  time.  Both  strategies  B  and  C  require  the  use  of  collaborative  infrastructure  to 
enable  the  open  business  model  that  would  be  most  likely  to  produce  these  real  options 
values.  Strategy  C,  in  fact,  relies  most  heavily  on  collaboration  to  enable  the  kinds  of 
benefits  in  rapid,  spiral  acquisition  with  greater  competition  and  innovation  from  smaller 
players. 

Summary 

IWS  systems  that  were  developed  in  a  closed,  proprietary  model  have  performed 
well  and  provide  substantial  returns.  However,  a  new  paradigm  is  required  to  maintain 
military  superiority  and  wage  information-age  warfare.  Through  open-system  development 
and  open-business  models,  benefits  such  as  reusable  code,  lower  maintenance-upgrade 
costs,  and  greater  vendor  flexibility  in  supporting  system  modules  could  be  derived. 
Moreover,  the  Navy  can  leverage  new  technology  by  quickly  adopting  it  to  military  needs. 

Significant  investments  are  required  for  the  infrastructure  necessary  to  enable  all 
parties  (acquirers,  users,  developers)  to  collaborate  easily  and  effectively  in  the  new  open- 
business  model.  Analytical  tools  are  also  required  to  track  performance  of  the  multiple 
parties  involved  in  the  development,  acquisition  and  use  of  new  system  capabilities,  in 
conjunction  with  the  ability  to  adjust  options  models  as  uncertainties  and  risks  are  resolved 
over  time.  Performance  measurement  systems  (i.e.  KVA  performance  accounting  software) 
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and  predictive,  forecasting  software  programs  plus  risk  certification  training  provide 
additional  analytic  support  to  achieve  IWS  systems  acquisition  strategies. 


Appendix  A.  Case  Study  Notes 

Methodology 

Case  Study  Approach 

The  learning-time  approach  to  KVA  is  used  to  conduct  sub-process  scenario 
analyses  of  “As  Is,”  “Radical  1 ,”  and  “Radical  2.”  Core  elements  of  KVA  (such  as  “time-to- 
learn,”  “number  of  personnel  involved,”  and  “times  fired”)  produce  a  ratio  of  knowledge 
capital  (ROK)  resident  in  each  process.  ROK  derives  a  common  unit  of  measurement  for 
each  sub-process  within  the  track-management  process. 

Data  Collection 

Collecting  data  to  conduct  KVA  analysis  was  difficult  due  to  the  complex  nature  of 
track-management  procedures  in  the  CIC.  Outputs,  learning  time  and  touch  time  of  the 
many  sub-processes  that  comprise  the  entire  procedure  are  not  generally  collected  or 
retained.  Also,  training  times  and  required  OJT  are  targeted  at  specific  watch  stations  rather 
than  at  specific  processes  within  the  Navy.  Consequently,  data  was  derived  through 
numerous  interviews  with  SMEs  and  review  of  Personal  Qualification  Standards  (PQS). 
Multiple  SMEs  were  contacted  to  collect  an  aggregated  sample. 

“To  Be”  Data 

Analysis  based  of  situations  that  SMEs  identified  as  optimal  areas  where  open 
architecture  could  provide  value  to  the  operator.  In  addition,  technical  and  legal  issues  of 
the  “To  Be”  scenario  were  not  assessed. 


Discussion  of  Basic  Assumptions  Used  in  Calculations 

Cost  and  Revenue  Data 

■  Calculations  based  on  a  ship  performing  SA  and  track  management  processes  24 
hours  a  day,  7  days  a  week  operating  an  average  of  35  weeks  per  year. 

■  Net  present  value  in  total  revenue  estimates  is  based  on  a  30-year  system-life 
expectancy  of  ship. 

■  A  contractor-margin  approach  was  used  to  generate  surrogate  comparable  revenue 
estimates.  Contractor  margin  is  defined  as  the  amount  market  place  would  pay  a 
group  of  contractors,  with  levels  of  knowledge  comparable  to  the  existing  team,  to 
perform  the  activities  of  the  track  management  team  (e.g  the  margin  over  current 
Navy  costs  for  the  track  management  team).  This  market-comps  approach  was 
used  because  there  were  no  commercial  processes  directly  comparable  to  track 
management  activities  in  the  military.  Future  studies  would  allow  a  wider  range  of 
potential  commercially  comparable  processes  to  be  assessed  for  comparability/fit 
revenue  estimates. 
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■  Surrogate  comparable  revenue  estimates  are  conservative  given  that  revenue 
caiculations  were  based  on  six  people  in  the  SA,  track-management  process.  In 
reality,  more  peopie  may  be  involved  in  this  process. 

New  Development/Retrofitting  DD(x) 

■  PV  Asset  New  Deveiopment:  Calcuiated  by  translating  houriy  revenue  of  the 
combined  SSDS/AEGIS  “AS-IS”  estimates  to  yearly  revenue,  calculating  the  present 
vaiue  of  the  yearly  revenue  based  on  a  30-year  lifecycle,  and  assuming  retrofitting  on 
ail  current  84  Aegis  destroyers/cruisers. 

■  PV  Asses  Retrofitting:  Calcuiated  by  transiating  houriy  revenue  of  the  combined 
SSDS/AEGIS  RADICAL  estimates  to  yeariy  revenue,  calcuiating  the  present  vaiue  of 
the  yeariy  revenue  based  on  a  30-year  lifecycle,  and  assuming  retrofitting  on  all 
current  84  AEGIS  destroyers/cruisers. 

■  Cost  to  Execute:  Based  on  estimation  from  SMEs  and  refining  based  on  historical 
costs  of  AEGIS.* 

■  Operations/Maintenance  Costs:  Based  on  historical  costs  of  AEGIS  and  scaled  down 
to  account  for  open  architecture. 

■  Timing:  Based  on  estimates  by  SMEs. 

Leave  and  Layer/AEGIS 

■  PV  Asset:  Calculated  by  translating  hourly  revenue  of  the  combined  SSDS/AEGIS 
estimates  to  yearly  revenue,  calculating  the  present  value  of  the  yearly  revenue 
based  on  a  30-year  lifecycle,  and  taking  into  account  current  placement  on  84 
destroyers/cru  isers . 

■  Cost  to  Execute:  Based  on  $5  billion  development  costs  and  scaled  according  to 
increases  in  knowledge  units. 

■  Operations/Maintenance  Costs:  Based  on  historical  costs  of  AEGIS  and  scaled  down 
to  account  for  open  architecture. 

■  Timing:  Based  on  estimates  by  SMEs. 
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Use  of  Data  in  Case  Study  Calculation 


Item 

Definition 

Comments 

No.  of 

Personnel 

Sailors  and  officers  involved 
in  performing  sub-process. 

— 

Actions  per 

Hour 

Times  each  sub-process 
acted  upon  by  watch- 
stander. 

•  Actions  predicated  on  amount  of  contacts  (air  and  surface)  encountered  during  a 
typical  hour  within  the  CIC.  Each  contact  must  be  acted  upon. 

•  Estimates  based  on  a  typical,  six-month  deployment. 

•  Number  of  contacts  based  on  average  of  open-ocean  transit  and  operations  in 
littorals. 

Actual  Work 

Time  (AWT) 

Specific  amount  of  time 
required  to  accomplish  action 
every  time  sub-process  acted 
upon. 

•  Data  captured  in  hourly  units. 

•  Although  actions  require  only  seconds,  category  captures  data  in  hours  to 
maintain  continuity  of  units  of  time  throughout  analysis. 

Total  Work 

Time 

Total  amount  of  time  each 
sub-process  acted  upon 
within  an  hour. 

•  Formula=  “AWT’  x  “Actions  per  Hour” 

•  Analysis  in  hourly  units:  when  “Total  Work  Time”  for  each  of  sub-processes 
added  together  for  each  of  the  watch  stations,  total  aggregate  should  remain 
below  1 .0.  If  total  exceeded  1 .0,  calculations  are  incorrect. 

Actual  Learning 
Time  (ALT) 

Total  amount  of  time  required 
to  learn  given  sub-process. 

•  Learning  time  can  be  an  aggregate  of  formal  schools,  distance  learning,  on-the- 
job  training  (OTJ)  or  any  other  training  experience  that  falls  under  definition  of 
“learning.” 

•  For  this  case,  comprised  of  formal  school  training  and  OJT  provided  aboard  ship. 

•  Basic  assumptions  to  ensure  consistent  estimates  from  SMEs: 

a.  Officer-SSDS 

Individual  completing  initial  officer  training  with  no  prior  SSDS  platform  experience.  It 
was  also  necessary  to  determine  formal  schools  represented  by  this  category.  While 
each  school’s  duration  is  considerably  longer  than  hours  represented  in  the  “ALT” 
category,  estimates  based  on  the  aggregated  amount  of  time  devoted  to  teaching 
given  sub-process  from  each  school:  SSDS  Basic  Operator  Course  of  Instruction, 
SSDS  Advanced  Operator  Course  of  Instruction,  and  SSDS  Warfare  Operator 

Course  of  Instruction. 

b.  Officer-AEGIS 

Individual  completing  officer  training  with  no  prior  AEGIS  platform  experience.  It  was 
also  necessary  to  determine  formal  schools  represented  by  this  category.  While 
each  school’s  duration  is  considerably  longer  than  the  hours  represented  in  the 
“ALT”  category,  estimates  based  on  aggregated  amount  of  time  devoted  to  teaching 
the  given  sub-process  from  each  school:  AEGIS  Training  Course,  SWOS  TAO 

School,  and  TAO  Simulator  Training. 

c.  Enlisted-SSDS 

Individual  completed  boot  camp  with  no  prior  SSDS  platform  experience.  It  was  also 
necessary  to  determine  formal  schools  represented  by  this  category.  While  each 
school’s  duration  considerably  longer  than  hours  represented  in  the  “ALT”  category, 
estimates  determined  based  on  aggregated  amount  of  time  devoted  to  teaching  the 
given  sub-process  from  each  school:  OS  “A”  School,  SSDS  Basic  Operator  Course 
of  Instruction,  SSDS  Advanced  Operator  Course  of  Instruction,  and  SSDS  Warfare 
Operator  Course  of  Instruction  (E5  and  above). 

d.  Enlisted-AEGIS 

Individual  completed  boot  camp  with  no  prior  AEGIS  platform  experience.  It  was  also 
necessary  to  determine  formal  schools  represented  by  this  category.  While  each 
school’s  duration  is  considerably  longer  than  hours  represented  in  the  “ALT” 
category,  estimates  based  on  aggregated  amount  of  time  that  devoted  to  teaching 
given  sub-process  from  each  school:  OS  “A”  School  and  AEGIS  Console  Operator 
Course. 

Rank  Order 

An  ordinal  ranking  of  sub¬ 
processes  provides  a  means 
to  ensure  the  “ALT”  estimates 
are  reliable  and  as  accurate 
as  possible. 

•  Allowing  SMEs  to  rank/order  each  of  the  sub-processes  (1  being  the  least 
complex),  outside  the  context  of  units  of  time,  facilitates  a  mathematical 
correlation  achieved  between  “Rank  Order”  and  “ALT”  categories. 

•  If  correlation  is  .80  or  higher,  “ALT”  numbers  can  be  considered  an  accurate 
reflection  of  the  sub-process’s  complexity. 

•  If  correlation  is  below  .80,  “ALT”  estimates  should  be  closely  scrutinized  and 
possibly  reevaluated  after  providing  a  better  explanation  of  the  “ALT”  components 
to  the  SMEs. 

Percent 

Percent  of  automation  for 

•  Captures  knowledge  embedded  within  the  IT  so  that  it  can  be  accounted  for  in 
later  calculations.  Automation  is  defined  as  the  amount  of  the  sub-process  that  is 
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Information 

Technology 

(%IT) 

each  sub-process. 

performed  by  information  technology  systems  and  does  not  require  the  actions  of 
an  operator. 

•  Each  sub-process  is  represented  by  a  percentage  between  0  and  100.  A  number 
of  100%  indicates  sub-process  is  completely  automated  and  does  not  require  a 
watch  stander  to  accomplish  any  portion  of  the  task. 

•  If  number  is  0%,  no  automation  exists,  and  the  watch-stander  completes  the  entire 
sub-process  manually.  Numbers  falling  between  extremes  are  estimates  based 
on  SME  observations  and  experience. 

Total  Learning 
Time  (TLT) 

Provides  total  time  required 
to  learn  sub-process, 
including  that  learning  time 
which  is  resident  within  the 

IT  system. 

•  Determined  by  “Actual  Learning  Time”  by  the  “Percent  Information  Technology” 
category. 

•  Formula  =  ALT/(1-%IT). 

•  For  instance:  If  it  takes  2  hours  to  learn  a  system  that  is  50%  automated,  then  the 
total  learning  time  for  that  system  (to  include  the  learning  time  that  is  embedded  in 
the  system  itself)  would  be  4  hours. 

Numerator 

Revenue  generated  by 
knowledge  required  to 
perform  sub-process. 

•  Revenue  allocated  to  the  amount  of  knowledge;  amount  of  knowledge  resident  in 
sub-process. 

•  Formula=  “Number  of  Personner  x  “Actions  per  Hour''  x  “Total  Learning 
Time" 

Denominator 

Cost  associated  with 
producing  sub-process 
output. 

o  Formula  =  “Number  of  Personnel"  X  “Actions  per  Hour"  X  “Actual  Work 
Time" 

Return  on 

Knowledge 

(ROK) 

Represents  how  well 
knowledge  assets  in 
organization  are  distributed 
based  upon  cost  and  value 
each  provides. 

•  With  every  sub-process,  there  is  a  cost  and  revenue  (or  value)  associated  with 
generating  an  output.  While  these  costs  and  values  are  captured  in  the 
“Numerator”  and  “Denominator”  categories,  there  needs  to  be  a  way  to  quantify 
the  knowledge  embedded  within  an  IT  system. 

•  ROK’s  can  be  compared  within  a  process  to  help  determine  if  knowledge  assets 
are  being  used  in  an  efficient  manner;  if  automation  could  be  inserted  to  improve 
outputs;  and  if  processes  should  be  changed  to  promote  efficiencies. 

•  A  low  ROK  does  not  automatically  assume  a  process  is  inefficient  or  in  need  of 
automation,  but  rather  is  an  indicator  that  a  sub-process  may  need  further 
analysis  to  determine  if  it  is  using  its  knowledge  assets  in  an  efficient  manner. 

Variability  Report 

Average  Work  Time:  5%  Variability — The  time  it  takes  to  complete  each  action  is 

relatively  stable  with  little  or  no  variability. 

Average  Learning  Time:  5%  Variability — Estimates  regarding  average  learning  time  are 

based  on  the  time  it  takes  an  average  person  to  learn  each 
task,  hence,  low  variability. 


Price:  Assume  60%  of  the  time  the  position  is  priced  at  the  average 

of  the  low  and  high  estimates.  The  remaining  time  is  split  20- 
20  between  the  low  and  high  values.  A  custom  distribution  is 
utilized  to  fulfill  these  requirements. 


Watch  Station 

Cost  Ranqe  (per  hour) 

Tactical  Action  Officer  (TAO) 

Anti-air  Warfare  Coordinator  (AAWC) 
Surface  Warfare  Coordinator  (SUWC) 
Combat  Systems  Coordinator  (CSC) 
Tactical  Information  Coordinator  (TIC) 
Identification  Supervisor  (IDS) 

$85  to  $105,  $80.00  to  $1 10.00 
$75  to  $90,  $72.00  to  $92.00 
$75  to  $90,  $72.00  to  $92.00 
$78  to  $95,  $69.00  to  $89.00 
$70  to  $80,  $65.00  to  $85.00 
$70  to  $80,  $63.00  to  $83.00 

Note:  Prices  provided  by  commercial  vendors. 


Cost:  Assumes  60%  of  the  time  the  position  is  filled  by  a  person  with 

the  assumed  pay  grade.  The  remaining  time  is  split  20-20% 
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between  a  person  with  one  rank  higher  and  one  rank  lower.^® 
A  custom  distribution  is  utilized  to  fulfill  these  requirements. 


Watch  Station 

Years  of  Service 

Cost 

Tactical  Action  Officer  (TAO):  (0-5) 

between  10-18 

$38 

Anti-Air  Warfare  Coordinator  (AAWC):  (0-4) 

between  8-16 

$34 

Surface  Warfare  Coordinator  (SUWC):  (0-3) 

between  6-14 

$30 

Combat  Systems  Coordinator  (CSC):  (E-8) 

between  10-18 

$27 

Tactical  Information  Coordinator  (TIC):  (E-6) 

between  4-14 

$17 

Identification  Supervisor  (IDS):  (E-5) 

between  3-8 

$14 

Note:  Costs  calculated  by  averaging  monthly  salary  plus  sea  pay  for  the  assumed 

ranks  at  low/high  estimates  of  years  in  service. 


Actions  per  Hour:  A  triangular  distribution  with  min/max/most-likely  values  based 

on  calculations  from  the  following  numbers. 


Costai 

Open  Water 

Hiqh  Density 

Number  of  Contacts  per  Hour 

24-42 

12-30 

28-66 

Time  in  Location 

15% 

25% 

60% 

Note:  Data  provided  by  SMEs 


In  the  case  of  a  TAO,  assume  80%  of  the  time  the  position  is  filled  by  an  0-5;  in  the  case  of  an  IDS, 
assume  80%  or  the  time  the  position  is  filled  by  an  E-5. 
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Appendix  B.  KVA  Results  by  Watch  Station 


Tactical  Action  Officer  Anti -Air  Warfare  Coordinator  Surface  Warfare  Coordinator 


“AS  IS” 

“TO  BE” 

“RAD  1” 

“  RAD  2” 

“AS  IS” 

“TO  BE” 

“RAD  1” 

“  RAD  2” 

“AS  IS” 

“TO  BE” 

“RAD  1” 

“  RAD  2” 

CORRELATE 

Obtain  Link  Information 

3625% 

3450% 

3206% 

2717% 

5091% 

4907% 

4525% 

3979% 

2441% 

2304% 

2230% 

2044% 

Identify  “Same  Contact, Multiple  Track” 

-91% 

-91% 

2104% 

1778% 

-87% 

-87% 

2983% 

2619% 

-94% 

-94% 

1453% 

1329% 

Verify  Other  Track  Sources 

-94% 

-94% 

-95% 

-96% 

-92% 

-92% 

-93% 

-94% 

-96% 

-96% 

-96% 

-97% 

Correlate  Sub-Total 

1358% 

1290% 

1791% 

1512% 

1932% 

1860% 

2546% 

2233% 

497% 

465% 

476% 

431% 

TRACK 

Monitor  Suspect  Tracks 

-98% 

-99% 

-99% 

-99% 

-98% 

-98% 

-98% 

-98% 

-99% 

-99% 

-99% 

-99% 

Update  Tracks 

-97% 

-97% 

294% 

235% 

-96% 

-96% 

451% 

386% 

-98% 

-98% 

Ml% 

155% 

Update  GCCS-M 

-95% 

173% 

154% 

117% 

- 

- 

- 

- 

-97% 

85% 

79% 

65% 

Track  Sub-Total 

-98% 

-97% 

-85% 

-87% 

-97% 

-97% 

-50% 

-56% 

-97% 

-55% 

47% 

36% 

IDENTIFY 

Verify  IFF  signal 

645% 

610% 

561% 

463% 

938% 

901% 

825% 

716% 

408% 

381% 

366% 

329% 

Verify  EW  emissions 

-89% 

-89% 

-90% 

540% 

-89% 

-91% 

-92% 

827% 

-92% 

-93% 

-93% 

387% 

Verify  Point  of  Origin 

-98% 

3450% 

3206% 

2717% 

-97% 

4907% 

4525% 

3979% 

-99% 

2304% 

2230% 

2044% 

Match  Against  ATO 

-98% 

3450% 

3206% 

2717% 

-97% 

4907% 

4525% 

3979% 

- 

- 

- 

- 

Match  Against  CommAir  Profile 

831% 

788% 

727% 

604% 

1198% 

1152% 

1056% 

920% 

- 

- 

- 

- 

Match  Against  Intel  Information 

-97% 

-97% 

-97% 

3813% 

-96% 

-96% 

-96% 

5565% 

-98% 

-98% 

-98% 

2878% 

Examine  Kinematic  Data 

-96% 

-96% 

-97% 

-97% 

-95% 

-95% 

-95% 

-96% 

-97% 

-98% 

-98% 

-98% 

Obtain  Visual  ID 

-100% 

-100% 

-100% 

-100% 

-100% 

-100% 

-100% 

-100% 

-100% 

-100% 

-100% 

-100% 

Conduct  Verbal  Query 

-99% 

-99% 

-99% 

-99% 

-99% 

-99% 

-99% 

-99% 

-99% 

-99% 

-99% 

-99% 

Identify  Sub-Total 

55% 

152% 

134% 

596% 

84% 

197% 

174% 

689% 

-95% 

-93% 

-93% 

-26% 

RELAY 

Send  Over  Links 

-91% 

-91% 

-92% 

-93% 

-87% 

-87% 

-88% 

-90% 

-94% 

-94% 

-94% 

-95% 

Discuss  Picture  with  Battle  Force  Units 

-99% 

-99% 

-99% 

-99% 

-98% 

-98% 

-98% 

-99% 

-99% 

-99% 

-99% 

-99% 

Relay  Sub-Total 

-98% 

-100% 

-100% 

-100% 

-98% 

-98% 

-98% 

-98% 

-99% 

-100% 

-100% 

-100% 

TOTAL 

306% 

347% 

389% 

558% 

354% 

400% 

451% 

640% 

9% 

13% 

17% 

64% 

Figure  B-1.  AEGIS  ROI  Estimates 


Combat  Systems  Coordinator 

Tactical  Information  Coordinator 

Identification  Supervisor 

“AS  IS” 

“TO  BE” 

“RAD  1”  “  RAD  2” 

“AS  IS” 

“TO  BE” 

“RAD  1” 

“  RAD  2” 

“AS  IS” 

“TO  BE” 

“RAD  r 

’  “RAD  2” 

CORRELATE 

Obtain  Link  Information 

1040% 

1034% 

937% 

743% 

5012% 

4889% 

4510% 

4146% 

5921% 

5772% 

5312% 

4870% 

Identify  "Same  Contact,  Multiple  Track" 

-97% 

-97% 

591% 

462% 

-87% 

-88% 

2973% 

2731% 

-85% 

-85% 

3508% 

3213% 

Verify  Other  Track  Sources 

-98% 

-98% 

-98% 

-99% 

-92% 

-92% 

-93% 

-93% 

-91% 

-91% 

-92% 

-92% 

Correlate  Sub-Total 

346% 

344% 

493% 

383% 

1901% 

1853% 

2537% 

2329% 

2257% 

2199% 

2996% 

2743% 

TRACK 

Monitor  Suspect  Tracks 

-100% 

-100% 

-100% 

-100% 

-98% 

-98% 

-98% 

-98% 

-97% 

-98% 

-98% 

-98% 

Update  Tracks 

-99% 

-99% 

65% 

34% 

-96% 

-96% 

449% 

405% 

-96% 

-96% 

544% 

492% 

Update  GCCS-M 

_ 

_ 

_ 

_ 

_ 

_ 

_ 

_ 

_ 

_ 

Track  Sub-Total 

-99% 

-99% 

-94% 

-96% 

-97% 

-97% 

-5% 

-13% 

-97% 

-97% 

11% 

2% 

IDENTIFY 

Verify  IFF  signal 

128% 

127% 

107% 

69% 

922% 

898% 

822% 

749% 

1104% 

1074% 

982% 

894% 

Verify  EW  emissions 

-96% 

-96% 

-97% 
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Figure  B-2.  AEGIS  ROI  Estimates  (cont.) 
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Figure  B-3.  SSDS  ROI  Estimates 
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Figure  B-4.  SSDS  ROI  Estimates  (cont.) 
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Appendix  C.  Real  Options  Analysis 
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Table  C-2.  KVA  Analysis  with  Monte  Carlo  Risk-based  Simulations 
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Table  C-3.  Strategy  B:  Real  Options  Analysis  Results 


Table  C-4.  Strategy  C:  Real  Options  Analysis  Results 
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Introduction 

That  management  adds  value  to  organizations  is  one  of  the  unquestioned  truisms  of 
business,  government,  military,  and  any  other  multi-member  organization.  The  question  left 
largely  unanswered  is,  “How  much  value  does  management  add  to  an  organization?”  The 
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central  research  focus  of  this  study  is  to  establish  a  method  for  objectively  measuring  the 
value  management  adds  to  an  organization. 

Determining  the  value  added  by  management  becomes  particularly  important  as 
Navy  acquisition  managers  deal  with  increasingly  complex,  open  business  models  that 
engage  many  more  participants  in  the  development  and  implementation  of  products  and 
services  to  support  warfighters.  There  is  a  recognition  that  emerging,  increasingly  complex 
acquisition  environments  require  more  direction,  collaboration,  and  control  to  achieve  the 
reductions  in  costs  as  well  as  the  increases  in  warfighting  effectiveness  that  initiatives,  such 
as  the  Open  Architecture  (OA)  system  acquisition  and  development  framework,  promise. 
The  increasing  burden  on  management  in  such  environments  largely  derives  from  the 
amount  of  complexity  managers  must  deal  with  by  attempting  to  mitigate  risks,  improve 
predictions,  and  exercise  the  control  and  oversight  necessary  to  be  successful. 

Determining  the  value  added  by  management  Is  Important  because: 

■  Managers  need  to  know  how  well  they  are  performing  in  more  complex  and 
demanding  environments  such  as  OA  and  open  business  models. 

■  Those  who  evaluate  management  performance  need  a  common  metric  to  gauge  the 
degree  to  which  managers  are  succeeding  or  failing. 

■  Poor  performance  by  management  threatens  delivery  time,  acquisition  costs,  and 
capability  of  acquired  products  and  services. 

■  A  consistent  and  objective  way  to  evaluate  management  provides  historical 
performance  data  that  leads  to  more  precise  risk  estimates  (instead  of  uncertainty- 
based  guesses). 

■  Including  management  performance  in  the  overall  assessment  of  organizational 
performance  provides  a  more  complete  picture  of  how  well  an  organization  is 
performing. 

Objectivity  Needed 

Solving  this  problem  requires  a  new  and  more  objective  approach  to  measuring  the 
value  added  by  management  than  previous  efforts.  Subjective  approaches  and  approaches 
based  on  corporate-level  residuals — ^which  presumably  are  a  result  of  management 
activity — will  not  provide  the  kinds  of  precision  necessary  to  allocate  value  (e.g.,  revenue, 
outputs)  to  individual  managers  in  proportion  to  the  amount  they  contribute  to  the 
organization’s  value  stream. 

Management  literature  is  replete  with  the  characteristics,  motivations,  and  general 
mantras  regarding  what  it  takes  to  develop  and  sustain  great  managers.  There  are  even 
more  articles,  books,  motivational  tapes  and  videos  on  what  it  takes  to  become  a  great 
leader.  Undoubtedly,  these  suggestions  can  lead  to  great  management.  However,  without  a 
way  to  partition  the  actual,  countable  contributions  that  management  makes  to  the  corporate 
bottom  line  or  general  organizational  success,  there  will  be  no  way  to  tell  which  suggestions 
really  work  and  which  add  little  or  no  marginal  value. 
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Research  Focus 


This  research  will  focus  on  making  the  case  for  a  method  to  objectively  measure  the 
value  added  by  management  activities.  We  will  review  the  general  issues  involved  in 
developing  a  theoretically  sound,  rigorous,  and  pragmatic  approach  to  estimating  the  value 
added  by  management.  The  paper  will  conclude  with  an  example  of  how  this  approach 
would  work  in  an  open  acquisitions  environment. 

Management  "Dark  Matter” 

Dark  matter,  in  the  physics  sense,  is  largely  unobservable — albeit  critical  to 
understanding  the  physics  of  the  universe.  The  dark  matter  of  management  has  also  been 
largely  unobservable  in  the  outputs  of  the  core  processes  of  an  organization,  although  it  is 
also  critical  to  understanding  the  functioning  of  an  organization.^^  These  mysterious 
elements  are  the  managing  activities  that  guide  the  organization  toward  its  future  and  are 
assumed  to  be  associated  with  the  market  outcomes  (e.g.,  increases  or  decreases  in 
revenue,  military  capability)  managers  attempt  to  influence. 

Management  activities  contribute  directly  to  the  outputs  of  the  processes  that  are 
being  managed,  but  they  also  involve  the  use  of  managers’  creative  insights  when  they 
attempt  to  predict  the  future,  create  potential  pathways  to  accomplish  the  predictions,  and 
control  for  future  risks  and  uncertainties.  Those  activities  that  are  uniquely  associated  with 
management  involve  the  creative  use  of  decision  heuristics  based  on  managers  implicit 
knowledge  accumulated  over  years  of  experience,  training  and  education.  We  label  this 
creative  aspect  as  management  “dark  matter.”  This  management  “dark  matter”  has  largely 
been  assumed  to  be  critical  to  the  duties  of  a  manager  but  has  not  been  objectively 
measured  as  researchers  attempted  to  account  for  the  value  added  by  management. 

These  heuristics  from  management’s  dark  matter  are  not  algorithmically  definable 
and  previously  have  been  subsumed  within  the  standard  knowledge  value  added  (KVA) 
methodology  in  gross  estimates  of  the  overall  output  of  an  organization  at  a  given  point  in 
time.  Basic  KVA  theory  is  designed  specifically  for  all  processes,  activities  that  are 
algorithmically  definable  a  priori,  i.e.,  for  processes  with  predetermined  outputs  (Housel  & 
Kanevsky,  1995).  KVA  posits  an  analytic  tautology  that  assumes  that  historical  outputs  of 
an  organization  at  a  given  point  in  time  can  be  described  in  common  units  and,  therefore, 
can  be  counted  in  absolute  terms.  Further,  the  total  revenue  of  a  commercial  organization  is 
equivalent  to  the  total  number  of  common  units  of  output  at  a  given  point  in  time.  The  theory 
must  now  be  expanded  to  account  for  those  managing  activities  that  involve  creative 
attempts  to  resolve  risk,  uncertainty  and  take  advantage  of  upside  potential. 

To  maintain  the  historical  and  analytic  context  of  KVA,  we  will  provide  a  way  to  count 
the  management  dark  matter  activities  within  estimates  of  the  overall  output  of  the 
organization  at  a  given  point  in  time.  This  is  reasonable  given  the  fact  that  previous  dark 
matter  activities  can  be  found  in  current  process  outputs;  they  usually  take  the  form  of 
changes  to  those  process  descriptions  that  lead  to  new  or  changed  outputs. 


Notionally,  the  grey  (dark)  matter  of  the  brain  also  is  where  original  thinking  occurs. 
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For  instance,  in  an  open-business-model  environment,  if  an  acquisition  manager 
suggests  that  a  vendor  change  some  aspect  of  his/her  core  processes  to  ensure  that  a 
project  stay  on  schedule,  and  the  vendor  responds  by  changing  his/her  process  to  mitigate 
the  perceived  risk,  then  the  acquisition  manager’s  dark  matter  will  eventually  be  manifested 
in  the  changed  process.  This  example  assumes  that  the  acquisition  manager  and  vendor 
are  part  of  a  virtual  organization  working  collaboratively  toward  a  final  outcome,  e.g.,  an 
integrated  weapons  system. 

In  the  ARCI  (Acoustic  Rapid  COTS  Insertion)  program,  an  open  architecture  (OA), 
open-acquisition  business  model  was  used  to  rapidly  transition  new  technologies  from  the 
Advanced  Submarine  Technology  Office  (ASTO)  housed  within  the  Program  Executive 
Office-Integrated  Weapons  Systems  (PEO-IWS)  and  the  ARCI  program  (housed  within  a 
different  program  office,  i.e.,  PEO  Subs).  This  created  an  increase  in  the  complexity  of  the 
acquisition  cycle.  However,  it  also  allowed  the  ARCI  program  to  leverage  the  resources  and 
outputs  of  the  ASTO  program  when  it  created  an  inter-program  partnership.  The  OA,  open- 
business  model  provided  the  environment  that  created  this  opportunity  as  well  as 
contributed  to  the  increased  complexity.  In  this  environment,  the  ARCI  acquisition 
management  had  to  use  more  of  its  dark  matter  to  ensure  closer  collaboration  among  the 
various  parties  as  well  as  to  predict  and  mitigate  possible  uncertainties  and  risks  while 
leveraging  the  multiple  resources  of  the  “virtual”  organization. 

Accounting  for  the  value  added  by  acquisition  managers,  in  this  case,  is  more  than 
just  accounting  for  their  routine  process  contributions;  it  also  requires  accounting  for  their 
use  of  dark  matter  capabilities  to  influence  the  future  behavior  of  the  virtual  organization.  If 
the  acquisition  manager  is  really  “doing  his/her  job”  in  this  open  business  environment,  s/he 
will  constantly  be  using  this  dark  matter  to  attempt  to  control  risks  and  uncertainties  by 
predicting  the  future  actions  necessary  to  mitigate  same. 

This  manager  can  no  longer  be  content  to  simply  ensure  that  the  vendor  meets  the 
stipulated  contract  obligations.  In  the  new  open  environment,  the  acquisitions  manager  must 
now  coordinate  with  numerous  potential  contractors  throughout  the  life  cycle  of  the  system 
acquisition.  These  must  become  a  part  of  a  virtual  management  team  that  collaborates 
regularly  to  ensure  that  the  developing  (or  developed)  system  meets  the  needs  of  the 
warfighter.  This  new,  more  complex  environment  introduces  greater  opportunities  for  the 
manager  to  use  his/her  dark  matter  to  creatively  anticipate  and  solve  problems  while  also 
seeking  better  ways  to  satisfy  the  constantly  changing  warfighter  requirements. 

Difficult-to-track  Dark  Matter  Outputs 

This  dark  matter  aspect  of  his/her  job  has  been  more  difficult  to  track  in  objective 
terms  and  largely  has  been  relegated  to  anecdotal  descriptions  of  the  occasional  successes 
and  failures  in  the  use  of  dark  matter  capabilities.  These  managers’  job  descriptions  include 
directions  to  use  their  dark  matter  capabilities  to  control  for  risks,  such  as  potentially  slipping 
development  and  implementation  schedules.  However,  it  is  often  more  easy  for  them  to 
simply  focus  on  the  daily  routines  of  “putting  out  fires”  than  to  stretch  their  dark  matter 
capacities  to  innovate  when  necessary  to  mitigate  future  risks  and  capitalize  on  future 
opportunities — in  other  words,  to  increase  warfighting  capabilities. 

We  can  use  standard  KVA  methodology  to  track  the  outputs  of  this  dark  matter 
activity  over  time  since  it  has  been  assumed  to  be  a  part  of  the  organization’s  overall  output 
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at  a  given  point  in  time.  As  such,  it  is  assumed  to  be  a  required  part  of  the  manager’s 
standard  outputs  and,  thus,  can  be  counted  within  the  standard  KVA  methodology. 
However,  we  also  assume  that  this  dark  matter  output  will  have  some  influence  on  the 
marketplace.  Therefore  this  output  should  correlate  with  changes  in  the  market,  such  as 
increased  sales. 

Computing  Metaphor 

Another  way  to  describe  management  dark  matter  conceptually  is  by  using  a 
computing  metaphor  as  has  been  used  previously  to  describe  the  KVA  methodology 
(Kanevsky  &  Housel,  1998).  Using  a  computing  analogy,  managers  use  their  dark  matter 
capabilities  to  write  the  program  for  the  processes  they  manage.  “Writing”  the  program 
includes: 


■  decisions  they  must  make  during  the  course  of  a  given  time  period,  including: 
o  Deciding  which  “program”  to  use 

o  Deciding  how  to  allocate  resources  to  produce  given  outputs 

o  Sustaining  activities  that  maintain  his/her  network 

o  Forecasting  the  amount  of  outputs  desired  from  the  processes  he/she 
manages 

o  Creating,  sending,  and  receiving  messages 
o  Selecting  and  using  technology  to  support  management  activities 

Writing  the  program  is  how  managers  create  value  in  the  processes  they  manage. 
The  time  it  takes  to  write  the  program  to  produce  a  given  output  is  the  cost  of  managing. 

The  actual,  non-redundant  length  of  the  program  is  proportionate  to  the  value  of  the 
program  (Housel  &  Kanevsky,  1995).  More  complex  dark  matter  decisions  require  longer 
programs,  and  simpler  decisions  require  shorter  programs.  Writing  a  relatively  long  program 
that  Includes  significant  amounts  of  redundancy  and  does  not  execute  or  executes  poorly  is 
how  managers  increase  costs — relative  to  managers  who  write  shorter,  i.e.,  more  elegant, 
programs  that  produce  the  same  outputs.  This  can  be  likened  to  the  “art  of  good 
management.” 

Technology  can  support  managers  in  producing  their  dark  matter  outputs.  For 
example,  in  an  open  acquisitions  environment,  collaborative  technology  can  support  a 
manager  in  recording,  distributing,  and  receiving  messages.  However,  while  “managing 
technology”  may  execute  some  routine  management  activities,  there  is  no  program,  at  this 
point,  that  can  write  its  own  unique  program.  This  makes  the  position  of  the  managers 
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unique  because,  at  this  point  in  time,^®  it  is  their  ability  to  create  elegant  programs  that 
largely  determines  their  success  in  managing. 

The  ability  of  a  manager  to  write  elegant  programs  that  produce  the  desired  amount 
and  kinds  of  process  outputs,  largely  determines  his/her  value  added  to  the  enterprise.  If 
they  write  programs  that  predict  poorly,  managers  will  fail  to  allocate  resources  properly  and, 
thus,  fall  to  produce  the  desired  outputs.  This  will  be  evidenced  by  overall  lower  returns  on 
investment  (ROIs)  on  given  processes  due  to  relatively  poor  utilization  of  process  assets 
such  as  technology  and  employees. 

Elegant,  parsimonious,  and  precise  management  “programmers”  will  produce  the 
best  outcomes  over  time.  This  may  be  due  to  a  variety  of  reasons,  including:  employees  that 
are  happier  because  they  are  more  productive,  the  ability  of  management  to  respond  quickly 
to  changes  in  markets-environments,  and  higher  utilization  of  technology  assets  based  on 
more  optimized  process  designs.  There  may  be  a  variety  of  outcomes  due  to  more  elegant 
programs;  however,  regardless  of  outcome,  elegant  programs  will  result  in  higher  ROIs  in 
managing. 

Dark  Matter  Correlates  with  Market  Performance 

We  recognize  that  these  dark  matter  activities  are  meant  to  influence  the  future 
behavior  of  the  core  processes  of  organizations  to  achieve  the  management  goals 
established  for  the  organization.  As  such,  they  should  be  correlated  with  the  market 
performance  of  the  organization  in  terms  of  the  impact  on  outcomes  obtained  by 
organizations  (e.g.,  increased  capabilities,  winning  the  battle,  more  revenue,  or  other  forms 
of  value). 

The  seeming  conundrum  is  that  while  these  dark  matter  activities  are  designed  to 
influence  future  behavior  and  outcomes,  historically,  they  can  be  accounted  for  at  a  given 
point  in  time.  That  they  are  part  of  the  output  of  an  organization  at  a  given  point  in  time  is 
clear;  however,  they  cannot  be  tied  directly  to  the  current  outputs  of  the  organization  that 
produce  the  value  or  capabilities  that  markets  are  willing  to  purchase. 

For  example,  in  our  new  open  business  environment  in  which  all  parties  are  part  of  a 
collaborative  virtual  organization,  an  acquisition  manager  would  feel  free  to  suggest  a 
potential  risk  mitigation  strategy  to  a  contractor  because  he/she  assumes  that  the 
development  schedule  will  slip  due  to  the  loss  of  a  key  programmer.  With  the  required 
increase  in  collaboration  among  the  contractor  and  the  other  parties  involved  in  the 
development  and  deployment  of  a  new  integrated  weapons  system  (IWS),  precipitated  by 
the  use  of  an  OA  and  open-business-model  approach,  managers  of  every  group  must 
engage  in  greater  cooperation  across  organizational  boundaries.  To  be  successful,  they  all 
must  use  their  collective  dark  matter  capabilities  to  recognize  and  address  potential  risks 
and  opportunities. 

In  this  new  cooperative  environment,  whether  the  acquisition  manager’s  or  the 
contractor’s  dark  matter  activity  is  directly  responsible  for  changes  to  IWS  development 


Ray  Kurzweil  (In  the  Age  of  Spiritual  Machines)  argues  that  computers  will  have  the  intelligence  to 
write  their  own  unique  programs  within  the  next  50  years. 
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processes  cannot  be  known  unambiguously — e.g.,  “Was  it  the  acquisition  manager  or  the 
contractor  manager  that  was  responsible  for  the  changed  process  that  avoided  the  possible 
schedule  slippage?”  The  only  thing  that  is  unambiguous  is  that  both  have  produced  the 
outputs  from  their  collective  dark  matter  capabilities,  and  these  should  be  treated  as  coming 
from  a  single  management  entity.  They  should,  therefore,  be  less  inclined  to  point  fingers 
when  things  go  wrong  due  to  poor  use  of  management  dark  matter  and  more  inclined  to 
ensure  that  the  IWS  is  delivered  on  time,  with  the  capabilities  required  by  the  warfighter. 

In  addition,  the  cooperative  use  of  dark  matter  to  achieve  common  goals  will  ensure 
that  managers  make  the  adjustments  necessary  to  seize  opportunities  to  create  greater 
capabilities  in  the  IWS  when  deemed  necessary  by  the  warfighter.  Tracking  their  collective 
use  of  dark  matter  makes  it  possible  to  assess  the  contributions  of  the  new  collaborative 
cross-organizational  management  team.  There  is  nothing  quite  like  being  measured  on 
overall  performance  to  drive  home  the  need  for  cooperation  to  achieve  common  goals. 

Outputs  of  Dark  Matter 

If  we  assume  that  these  dark  matter  outputs  are  a  necessary  part  of  the  output  of  an 
organization — and  this  would  appear  to  be  the  case  in  terms  of  our  expectations  for  as  least 
part  of  what  managers  should  be  doing:  i.e.,  predicting  the  future  and  controlling  for  risk — 
then  it  follows  that  these  dark  matter  activities  of  management  are,  a  priori,  designed  to  be 
part  of  the  output  of  the  organization.  Indeed,  a  review  of  management  job  descriptions 
would  include  imperatives  to  “lead,  motivate,  plan  for  the  future,  control  risks,”  all  of  which 
we  are  labeling  dark  matter  activities. 

For  example,  if  a  manager  uses  his/her  creative  insight  to  suggest  that  the  product 
should  be  painted  green  as  well  as  red,  the  painting  process  eventually  must  be  changed  to 
reflect  the  prediction  that  the  new  color  for  the  product  will  sell  better  to  a  given  segment  of 
the  market. 

Or,  for  example,  when  an  acquisition  manager  senses  that  the  schedule  Is  about  to 
slip  because  a  subcontractor  has  gone  bankrupt,  he/she  must  predict  the  effect  of  this  risk 
and  develop  strategies  to  mitigate  the  problem,  such  as  maintaining  an  option  to  purchase 
the  service  from  another  developer.  Another  example  might  be  when  the  “manager”  (officer) 
of  a  warfighting  process  that  tracks  ships  in  a  congested  area,  such  as  the  Persian  Gulf, 
recognizes  that  something  is  amiss  even  though  the  track  information  appears  to  be  correct. 
His  intuition,  based  on  the  dark  matter  acquired  over  years  of  experience,  comes  into  play  in 
seeking  further  clarification  of  the  track  information  because  something  (his  intuition,  or  dark 
matter)  tells  him  all  is  not  well,  and  this  ship  may  not  be  “friendly”. 

Operationalizing:  The  Measurement  of  Dark  Matter 

We  assume  that  managers  constantly  acquire  knowledge  both  formally  and 
informally,  learn  from  that  knowledge,  and  incorporate  that  knowledge  in  their  decision¬ 
making  in  predicting  the  future  and  in  planning  to  control  for  risk  and  uncertainty  from  a 
constantly  changing  environment.  This  form  of  management  “dark  matter”  output  is 
manifested  largely  in  the  messages  that  managers  generate,  distribute  and  interpret  to 
predict  and  control  core  processes  to  accomplish  the  goals  of  the  organization.  Most  of  a 
manager’s  time  is  spent  in  creating,  sending,  receiving  and  interpreting  messages.  These 
messages  are  basic  descriptions  of  actions  the  manager  is  influencing  his  organization  to 
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take.  Some  of  these  actions  are  unambiguously  tied  to  actual  production  of  the  process  they 
are  managing  and,  thus,  are  captured  in  the  standard  KVA  methodology  within  the 
description  of  process  outputs. 

The  central  idea  is  that  we  can  “see”  the  manifestation  of  that  acquired  knowledge 
and  the  subsequent  managing  activities  via  the  networks  that  managers  and  information 
technology  (e.g.,  collaborative  technology)  use  to  create,  send,  and  receive  messages. 
These  messages  are  sent  to  and  received  from  the  employees  and  the  technology.^®  The 
actual  contents  of  the  messages  take  many  forms  and  fill  many  purposes — all  of  which  can 
be  described  in  the  common  descriptive  language  provided  by  the  KVA  approach  in  terms  of 
their  relative  complexity. 

The  challenge  is  to  develop  an  operationalization  of  management  dark  matter 
outputs  that  permits  an  objective  bifurcation  of  relevant  and  irrelevant  activities  without 
resorting  to  overly  subjective  interpretations  and  the  potential  biases  such  subjective 
judgments  introduce.  At  the  very  least,  the  goal  should  be  to  establish  an  unambiguous 
principle  for  categorizing  relevant  and  irrelevant  activities  that  is  defensible  without  resorting 
to  subjective  judgment.  This  bifurcation  will  be  worked  out  as  we  attempt  to  operationalize 
this  approach  over  time  with  actual  case  studies  and  empirical  research.^® 

Those  messages  that  are  unambiguously  tied  to  current  process  outputs  represent 
routine  management  outputs  that  are  measurable  in  the  standard  KVA  methodology.  Those 
management  messages  that  are  not  found  directly  in  current  outputs  of  the  process  are 
evidence  of  management  dark  matter  activity.  Regardless  of  their  semantic  content, 
because  we  can  observe  all  the  messages  and  estimate  their  complexity,  they  represent  a 
convenient  way  to  observe  and  measure  management  dark  matter  outputs. 

Management  Dark  Matter  Outputs  and  Job  Descriptions 

Top  executives  often  state  goals  in  broad  sweeping  terms  such  as  “We  will  move 
from  product  leadership  to  cost  leadership  within  the  next  three  years;  We  will  move  from  a 
proprietary,  closed  business  model  to  an  open  business  model  in  acquisitions.”  Such 


Future  research  may  establish  a  more  refined  measure  of  dark  matter  activity  and  outputs  by 
examining  the  networks  managers  create,  modify  and  use  to  receive  and  distribute  their  messages. 
These  networks  vary  in  terms  of  their  complexity:  the  messages  sent  and  received  also  vary  in 
complexity.  These  two  forms  of  complexity  can  be  described  in  terms  of  the  knowledge  required  to 
reproduce  them,  and  the  knowledge  can  be  described  in  common  units  using  the  KVA  approach. 

With  this  information,  we  can  measure  the  amount  of  knowledge  transferred  through  each  link  in  the 
network — including  the  knowledge  used  to  maintain  and  modify  the  network. 

^  The  basic  probiem  is  to  determine  what  aspects  of  this  management  dark  matter  are  reievant  to 
organizational  value  and  which  are  not  directly  relevant.  It  is  possible  to  semantically  interpret 
management  activities  as  relevant  or  irrelevant  to  the  organization.  For  example,  when  a  manager 
arranges  a  dinner  date  with  his/her  spouse,  it  is  very  likely  this  activity  was  irrelevant  to  achieving 
organizational  goals.  However,  when  we  examine  less  obvious  examples,  such  as  a  manager  musing 
about  whether  to  invest  in  a  new  technology  that  may  or  may  not  ever  be  purchased,  it  is  less  obvious 
how  this  activity  led  to  some  organizational  outcome. 
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broadly  stated  goals  must  be  translated  into  action  at  the  process  level  through  changes  in 
the  process  descriptions  (e.g.,  make  more  widgets  or  make  different  kinds  of  widgets). 

Management  job  descriptions  should  reflect  the  dark  matter  capabilities  necessary  to 
translate  such  broad,  sweeping  goals  into  operational  realities.  One  way  to  measure  dark 
matter  output  would  be  to  examine  the  job  descriptions  of  managers  and  ask  them  to 
estimate  how  many  messages  and/or  activities  they  generate  to  fulfill  each  aspect  of  the  job, 
as  well  as  how  often  they  do  this  within  a  given  time  period.  To  check  the  accuracy  of 
manager’s  self  reports,  we  can  check  actual  message  logs,  e-mails,  other  text-based 
historical  records  to  ensure  there  is  a  high  enough  level  of  reliability  among  the  estimates  to 
proceed. 

It  would  also  be  possible  to  ask  them  to  generate  a  more  precise  job  description  if 
the  original  is  not  sufficiently  reflective  of  what  they  really  do.  The  next  step  would  be  to 
separate  the  duties  that  involve  routine  daily  operations  and  those  that  are  more  reflective  of 
management  dark  matter  activities. 

Case  Example:  Ship  Tracking  Process 

There  are  approximately  250K  ships  at  sea  world-wide  at  any  given  point  in  time. 

The  US  military  needs  to  know  which  are  friendly  and  which  are  potential  foes.  The 
responsibility  for  tracking  these  ships  currently  rests  with  the  Coast  Guard.  The  tracking 
process  must  be  managed  by  experienced  Watch  Commanders  who  can  intuitively  sense 
when  a  given  track  is  incorrect  (Figure  1).  This  ability  cannot  be  automated  at  this  point  in 
time  and  is,  therefore,  indicative  of  dark  matter.  Watch  Commanders  must  be  able  to 
anticipate  when  risks  of  false  identification  of  tracks  may  be  present. 


Figure  1.  Ship  Tracking  Process 
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Ship  Tracking  Management’s  Dark  Matter  Activities 

The  track  manager  and  his  watch  commander  must  supervise  the  process  to  ensure 
that  timely  and  accurate  information  is  provided  to  various  DoD  entities  to  help  them  achieve 
situational  awareness  (SA).  The  primary  job  of  management  is  to  ensure  its  subordinates 
are  correctly  tracking  the  ship  activity.  One  method  by  which  members  of  management 
check  tracking  is  the  quality  control  step,  in  which  managers  must  sign  off  on  track  reports 
as  accurate. 

In  this  environment,  there  are  times  when  the  managers  intuitively  recognize  that  the 
track  information  may  not  be  correct — even  though  they  may  not  be  completely  aware  of 
why  they  have  made  such  judgments.  Their  intuitions  lead  them  to  ask  their  subordinates  to 
re-run  the  track  information  gathering  to  ensure  its  accuracy.  Their  use  of  dark  matter 
results  in  communications/messages,  to  subordinates  to  rerun  the  tracking  process  on  given 
suspect  targets.  As  such,  the  communications  are  observable  applications  of  their  dark 
matter  and  can  be  described  via  the  standard  KVA  methodology. 

The  times  when  intuitions  are  correct  or  incorrect  also  can  be  counted.  If  we  take  a 
conservative  attribution  strategy,  we  will  only  count  the  correct  ones,  those  ships  that  turned 
out  to  be  incorrectly  tracked  and  categorized,  as  part  of  management’s  dark  matter  output. 
However,  it  can  also  be  argued  that  to  achieve  a  given  yield  rate  for  suspected  tracks, 
management  must  have  some  level  of  false  positives;  therefore,  all  should  be  counted  as 
management  dark  matter  outputs.  In  either  case,  this  simple  case  example  enabled  us  to 
identify  and  count  those  dark  matter  outputs  of  management  which  in  turn  allowed  us  to 
allocate  value  to  those  activities. 

In  this  case  example,  we  have  also  demonstrated  that  when  a  simple  dark  matter 
capability  can  be  explicitly  described  (i.e.,  we  may  uncover  the  heuristic  decision-making 
process  that  managers  used  to  identify  the  potential  incorrect  tracks),  it  becomes  possible  to 
capture  this  dark  matter  capability  in  intelligent  computer  programs  such  as  expert  systems. 
This  is  the  basis  for  many  expert  systems  that  embed  management  dark  matter  decision 
capabilities  within  their  code.  The  new  tracking  system  developed  by  the  Office  of  Naval 
Research  not  only  automated  most  of  the  tracking  process  but  also  largely  replaced  the 
need  for  the  managers  to  use  their  dark  matter  to  identify  potential  incorrect  tracks.  The 
new  system  greatly  improved  the  capabilities  to  track  the  enormous  number  of  ships  that 
previously  were  tracked  primarily  by  human  operators.  It  also  is  capable  of  gathering 
additional  information  that  makes  use  of  managers’  intuition  about  possible  inaccurate 
tracking  largely  unnecessary. 

However,  the  track  process  designers  continue  to  require  the  presence  of 
management  to  sign  off  on  the  accuracy  of  tracks  while  ensuring  that  the  routine  tracking 
process  is  carried  out  as  expected.  If  management  becomes  complacent  about  its  dark 
matter  duties,  it  may  assume  that  the  new  system  is  so  accurate  that  management  members 
need  not  waste  their  time  trying  to  find  the  potentially  inaccurate  tracks. 

Because  dark  matter  activities  involve  prediction,  the  possible  removal  of  this 
management  dark  matter  capability  must  be  considered  seriously  because  one  correct 
prediction — i.e.,  identification  of  an  inaccurate  track  by  management  that  was  not  caught  by 
the  new  system — might  lead  to  identification  of  a  suspect  ship  that  is  carrying  weapons  of 
mass  destruction.  Dismissing  management  prerogatives  embedded  in  its  dark  matter 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  - 172  - 


capabilities  by  “automating”  them  must  be  carefully  considered  before  leadership  eliminates 
this  capability. 

Standard  KVA  and  Delta  Correlation  Approaches 

The  cascading  of  management  goals  to  lower  levels  of  management,  and  eventually 
to  the  process  level,  represents  the  relative  influence  of  managers  on  their  organizations. 
This  kind  of  influence  takes  time  to  become  a  reality  at  the  process  level.  Because  the 
response  of  the  market  to  these  kinds  of  goal-setting  messages  is  uncertain,  it  is  necessary 
to  measure  the  change  or  delta  in  value  (e.g.,  capabilities,  revenue)  of  the  organization  over 
time  as  the  goals  eventually  become,  or  fail  to  become,  reality  in  process  changes. 

We  must  acknowledge  the  likelihood  that  these  kinds  of  unique  management 
activities  do  indeed  influence  organizational  outcomes — as  has  been  argued  in  prior 
research  as  well  as  in  general  business  publications  about  the  role  and  value  of  leadership 
and  management.  This  assumption  leads  us  to  propose  that  changes  in  these  activities 
should  be  roughly  correlated  with  changes  in  market  outcomes. 

Because  making  a  causal,  one-to-one  connection  between  dark  matter  messages 
and  changes  in  core  processes  is  problematic  due  to  the  semantic-interpretation  problems 
mentioned  earlier,  we  will  not  attempt  to  causally  connect  the  two.  Because  a  pure  historical 
causal  relationship  cannot  be  established  between  the  dark  matter-market  outcome-process 
change  deltas,  we  will  need  to  track  the  correlation  among  the  three  over  time  to  better 
understand  the  relationship  between  deltas  in  management  dark  matter  activity  and  the 
presumed  corresponding  deltas  in  market  responses. 

This  logic  leads  to  two  measures  of  management  dark  matter  activity.  First,  we  can 
track  the  deltas  in  management’s  dark  matter  activity,  process  changes,  and  market 
outcomes  over  time.  Second,  using  standard  KVA  methodology,  we  can  account  for  the 
amount  of  dark  matter  outputs  as  a  percentage  of  the  total  outputs  of  a  process  and, 
consequently,  of  the  organization  at  a  given  historical  point  in  time.^^ 

As  a  first  cut,  we  can  aggregate  the  delta  of  all  managers’  dark  matter  activity  and 
correlate  it  with  market  outcome  deltas.  Individual  manager  dark  matter  output  deltas  can 
be  compared  within  the  organization  and  correlated  with  the  market  outcome  deltas  to  infer 
which  managers  have  had  the  most  influence  on  market  outcomes.  This  approach  extends 
standard  KVA  theory  to  the  largely  uncharted  realm  of  how  management  influences 
organizational  outcomes. 

Basic  Hypothesis 

This  formulation  of  the  problem  enables  us  to  test  some  basic  hypotheses  about  the 
correlation  among  dark  matter  outputs  and  organizational  outcomes.  One  hypothesis  would 


In  the  first  case,  we  can  use  standard  KVA  methodology  to  allocate  outputs  and  associated  value 
(e.g.,  revenue,  capability)  to  individual  managers  at  a  given  point  in  time.  In  the  second  case,  we  can 
measure  the  relative  changes  in  deltas  among  the  managers’  dark  matter  activity  and  the 
corresponding  changes  in  market  deltas. 
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be  that  there  should  be  a  much  higher  percentage  of  dark  matter  outputs  in  an  open 
business  environment  (in  which  the  OA  approach  is  used  to  develop  IWS  systems) 
compared  to  a  closed  business  model  approach  (in  which  proprietary  IWS  systems  are 
developed). 

Similarly,  in  the  corporate  sector,  the  amount  of  management  dark  matter  produced 
over  a  given  period  of  time  should  correlate  with  the  amount  of  revenue  generated  over  a 
period  of  time.  It  should  follow  that  mature  industries  (e.g.,  petroleum  extraction)  that  have 
largely  mitigated  risks  and  operate  within  a  relatively  stable  environment  in  which  demand  is 
largely  predictable  should  have  less  of  this  kind  of  dark  matter  than  those  operating  within 
largely  unstable  environments  (e.g.,  biotechnology). 

The  same  would  pertain  to  military  environments  in  which  there  have  been  periods  of 
stable  competition  (e.g..  Cold  War)  among  nations  compared  to  periods  when  competition 
among  entities  is  ill-defined  and  highly  volatile  (e.g..  Global  War  on  Terror).  In  the  current, 
military  environment  in  which  there  is  greater  instability,  there  is  a  larger  need  for  the 
production  of  management  dark  matter  to  predict  and  deal  with  uncertainties  and  risks. 

Hypothesis  Test:  ARCI  versus  Proprietary  Acquisition 

The  ARCI  case  example  provides  a  simplistic  test  of  this  hypothesis.  We  can 
compare  the  rough  orders  of  magnitude  estimates  of  dark  matter  outputs  in  both  proprietary 
and  open  environments  and  then  correlate  these  with  the  outcomes  for  development  of  IWS 
systems  in  both  environments.^^  The  hypothesis  is  that  the  OA,  open-business-model 
environment  of  ARCI  generated  significantly  more  dark  matter  output  than  the  proprietary 
model  of  system  acquisition  in  IWS. 

The  hypothesis  rests  on  the  basic  assumption  that  when  managers  are  successful  in 
setting  and  implementing  goals  to  affect  the  markets  within  which  they  operate,  markets 
respond  positively  and  the  converse  is  also  true. 

Through  changes  that  they  induce  in  their  organization’s  core  processes,  the 
commercial  market  responds  by  purchasing  more  of  their  products,  the  same  number  of 
higher  priced  products,  or  both  outcomes.  In  the  military  “markets,”  managers/officers  use 
their  dark  matter  to  anticipate  future  actions  by  adversaries  by  adding  capabilities  and 
improving  system  acquisition  cycle-times  to  respond  to  new  events  and  competitor’s 
strategies. 

However,  when  managers  increase  these  kinds  of  creative  activities  and  the  result  Is 
lower  revenue  or  reduced  capabilities,  it  can  be  said  that  management  reduced  or  failed  to 
increase  value.  The  delta  in  dark  matter  management  activities  from  one  period  to  the  next, 
in  response  to  anticipated  changes  in  the  market,  should  correlate  to  organizational 
performance  over  time. 


It  does  not  permit,  however,  the  translation  of  deltas  in  dark  matter  activity  over  time  into  absolute 
numbers  against  which  value  can  be  allocated. 
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It  does  not  follow,  however,  that  merely  increasing  the  number  of  dark  matter 
activities  (e.g.,  setting  goals,  trying  to  impiement  an  increased  number  of  goais)  wili  result  in 
increases  in  value.  These  goals  and  other  creative  management  activities  must  eventually 
be  translated  into  operational  changes  in  core  processes  to  affect  any  changes  in  company 
outputs  and,  therefore,  responses  to  changes  in  the  market.  Such  inevitability  provides  a 
rationaie  for  counting  management  dark  matter  activities  within  counts  of  the  total  output  of 
an  organization  at  any  given  point  in  time.^®  Clearly,  these  activities  are  present  and  can  be 
described  with  the  standard  KVA  approach  currentiy  used  to  describe  the  outputs  of  aii 
processes  in  common  units.  It  follows  that  the  standard  KVA  approach  may  be  extended  to 
triangulate  with  the  correlational  approach  in  estimating  the  effect  of  dark  matter  outputs  on 
organizationai  outcomes. 

ROI  on  Management  Dark  Matter 

Using  the  KVA  knowledge  metaphor  to  describe  outputs,  the  absolute  total 
contribution  of  managing  a  process  is  equivalent  to  the  total  amount  of  knowledge  required 
to  produce  and  interpret  all  management  messages  during  a  given  time  period.  The  amount 
of  time  a  manager  (and  supporting  management  technology)  spends  using  this  knowledge 
is  the  cost  of  management.  It  follows  that  a  manager  who  must  use  a  large  amount  of 
knowledge  to  process  messages  but  can  process  those  messages  quickly,  provides  a 
relatively  good  return  on  investment  in  managing.  Those  who  take  longer  to  produce  a 
similarly  complex  output  cost  more  and  provide  a  iower  reiative  return  on  investment. 

We  would  need  to  make  a  further  separation  of  messages  that  invoive  use  of  dark 
matter  from  those  that  do  not.  Following  this  separation,  we  would  estimate  the  amount  of 
complexity  in  each  message  set  using  the  learning  time  or  other  KVA  approach,  and 
estimate  the  total  amount  of  output  from  the  dark  matter  activity  (i.e.,  the  prediction-based 
messages).  This  resulting  value  would  be  included  in  the  total  output  for  a  given  period,  and 
value  would  be  allocated  proportionately  as  is  done  currently  in  the  KVA  method  for  normal 
process  outputs. 

We  can  use  standard  KVA  methodology  to  allocate  outputs  and  associated  vaiue 
(e.g.,  revenue,  capability)  to  individual  managers  at  a  given  point  in  time. 

Resolving  uncertainty  requires  that  managers  make  a  prediction  about  what 
demands  the  future  wiii  make  on  current  processes  in  terms  of  the  amount  and  kinds  of 
outputs  required  from  those  processes.  These  messages,  interpretations  of  messages,  and 
decisions  about  resolving  risk  and  uncertainty  can  be  observed  over  time  such  that  it  is 


A  very  crude  and  simple  test  of  this  hypothesis  would  be  to  compare  the  dark  matter  job  activities  of 
the  acquiring  manager  in  a  proprietary  environment  with  the  dark  matter  job  activities  of  managers  in 
an  OA,  open-management  environment.  The  number  of  messages  and  complexity  of  messages  used 
to  coordinate  acquisition  in  an  OA,  open-business-models  environment  would  be  much  greater  than 
for  managers  in  the  proprietary,  single-vendor  context.  This  would  assume  that  the  processes 
managed  have  about  the  same  level  of  complexity.  Relative  complexity  of  the  process  becomes  the 
index  against  which  the  amount  of  dark  matter  per  unit  of  complexity  is  produced. 
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possible  to  gather  historical  data  on  the  number  and  amount  of  complexity  of  these  dark 
matter  managing  activities.^'' 

So,  in  addition  to  the  routine  KVA  data  that  estimates  the  number  of  common  units  of 
output  from  algorithmically  definable  processes,  it  also  is  possible  to  count  the  number  of 
dark  matter  outputs  and  their  respective  complexity.  The  total  amount  of  output,  then,  is  a 
function  of  the  number  of  times  a  given  asset  produces  algorithmically  defined  outputs 
multiplied  times  the  amount  of  complexity  required  to  produce  the  outputs  plus  the  dark 
matter  outputs  that  occur  in  that  given  time  period.^® 

Past  research  on  the  value  of  management  and  leadership  provides  a  useful  review 
of  alternative  approaches  to  this  daunting  problem.  There  are  a  number  of  lines  of  research 
that  have  attempted  to  address  this  problem  indirectly  and  may  offer  some  insights  about  its 
many  varied  aspects.  Thus,  it  would  be  useful  to  review  the  strengths  and  weaknesses  of 
these  prior  attempts  to  resolve  this  difficult  problem. 

Prior  Approaches  to  Assessing  the  Value  of  Management 

The  literature  on  the  value  added  by  managing  includes  the  effect  of  corporate 
governance  on  firm  value,  the  drivers  of  value  (including  managing  activities),  the 
characteristics  of  great  leaders,  failures  of  management,  and  fund  management. 

This  review  of  previous  attempts  to  assess  the  value  that  management  adds  includes 
research  on: 

■  How  corporate  governance  affects  a  firm’s  valuation, 

■  What  management  characteristics  can  lead  to  improved  company  value, 

■  Fund-management  approaches  to  solving  the  problem,  and 

■  Attempts  to  solve  the  management  value  added  quantification  problem  using  a 

knowledge  framework. 

Corporate  Management/Governance  Value 

The  current  literature  discussing  the  effect  of  corporate  governance  on  management 
valuation  does  not  focus  on  direct  valuation  of  individual  managers  or  those  in  positions  to 


This  formulation  assumes  that  we  do  not  count  redundancy  in  the  dark  matter  activities — such  as 
exhorting  the  vendor  to  “work  harder”  every  day  during  the  observational  time  period.  We  would  count 
the  non-redundant  or  unique  dark  matter  outputs.  In  this  case,  we  might  only  count  the  exhortation  to 
work  harder  as  one  output  instead  of  one  output  repeated  many  times.  We  would  also  separate  out 
the  non-relevant  activities — ^for  exampie,  when  the  manager  calls  his  wife  to  arrange  a  dinner  date. 

Given  that  there  are  many,  if  not  infinite,  alternative  paths  for  the  organization’s  future,  it  is 
impossible  to  enumerate  all  the  potential  opportunities  foregone  by  selecting  a  specific  path.  It  is, 
therefore,  not  useful  (in  accounting  for  dark  matter  outputs)  to  attempt  to  look  backward  to  determine 
whether  managers  might  have  better  utilized  their  dark  matter  to  make  “better”  predictions.  This  type 
of  interpretation  must  be  left  to  the  subjective  judgment  of  the  leadership  of  the  organization. 
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perform  management  governance  activities.  Instead,  the  current  literature  revolves  around 
two  major  themes:  an  analysis  of  corporate  governance  and  its  reiation  to  firm  vaiue  as  well 
as  an  analysis  of  corporate  management  techniques  and  suggestions  on  quaiitativeiy 
increasing  management’s  vaiue. 

Literature  focused  on  corporate  governance/management  focuses  on  two  major 
themes:  addressing  whether  corporate  governance  affects  firm  vaiue  and  providing 
suggestions  on  how  to  quaiitativeiy  increase  the  vaiue  of  corporate 
governance/management — thus,  increasing  firm  value.  Literature  focused  on  the  second 
theme  generally  does  not  relate  techniques  directly  to  quantitatively  measured  increases  in 
firm  value,  but  instead  implies  that  the  techniques  will  increase  value  in  some  way. 

Corporate  Governance  and  its  Effect  on  Firm  Vaiue 

Black,  Jang,  and  Kim  set  out  to  answer  the  question:  “How  do  a  country’s  corporate 
governance  rules,  or  the  corporate  governance  practices  of  individual  firms  within  a  country, 
affect  overall  firm  value  and  performance?”  (Black,  Jang,  &  Kim,  2003,  p.  1).  This  study 
answers  the  question  by  developing  a  corporate  governance  index  for  526  Korean  public 
companies.  The  index  is  based  on  information  obtained  on  shareholder  rights,  board  of 
directors  in  general,  outside  directors,  audit  committee  and  internal  auditor  disclosure  to 
investors,  and  ownership  parity.  It  then  uses  multiple  regression  equations  to  explain  the 
variance  in  the  corporate  governance  index  compared  to  firm  value.  The  analysis  finds 
evidence  that  corporate  governance  is  an  important  factor  in  explaining  the  market  value  of 
Korean  public  companies.  Following  a  similar  model  Beiner,  Drobetz,  Schmid,  and 
Zimmerman  (2006)  establish  a  relationship  between  corporate  governance  quality  and  firm 


value. 


Beiner,  Drobetz,  Schmid,  and  Zimmerman,  in  their  work  “An  Integrated  Framework  of 
Corporate  Governance  and  Firm  Valuation,”  state  that: 

Recent  empirical  research  shows  evidence  of  a  positive  relationship  between 
the  quality  of  firm-specific  corporate  governance  and  firm  valuation.  Instead 
of  looking  at  one  single  corporate  governance  mechanism  in  isolation,  [this 
report]  constructs  a  broad  corporate  governance  index  and  [applies]  five 
additional  variables  related  to  ownership  structure,  board  characteristics,  and 
leverage  to  provide  a  comprehensive  description  of  firm-level  corporate 
governance  for  a  representative  sample  of  Swiss  firms.  To  control  for 
potential  endogeneity  of  these  six  governance  mechanisms,  [the  report 
develops]  a  system  of  simultaneous  equations  and  [applies]  three-stage  least 
squares  (SSLS).  [The]  results  support  the  widespread  hypothesis  of  a  positive 
relationship  between  corporate  governance  and  firm  valuation.  (2006, 
publication  abstract) 

This  work  shows  a  relationship  between  the  quality  of  specific  corporate  governance 
practices  and  the  value  of  a  firm,  but  does  not  allow  for  direct  valuation  of  individual 
managers.  It  primarily  supplements  the  intuitive  hypothesis  that  quality  affects  value. 

Klein,  Shapiro,  and  Young  (2005)  prove  a  similar  relationship  between  quality  and 
value  in  Canadian  firms,  but  add  the  additional  element  of  the  ownership  category.  In  their 
report,  the  researchers  analyze  the  relationship  between  firm  value  and  indices  of  effective 
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corporate  governance  in  Canadian  firms.  “The  results  indicate  that  corporate  governance 
does  matter  in  Canada.  However,  not  all  elements  of  measured  governance  are  important, 
and  the  effects  of  governance  do  differ  by  ownership  category”  (Klein,  Shapiro,  &  Young, 
2005,  publication  abstract).  The  results  of  this  study  establish  the  link  between  quality  of 
governance  and  value. 

An  additional  work  with  similar  results  is  Hsiu-I  Ting’s  (2006).  In  this  work.  Ting 
investigates  the  207  IPO  companies  encompassing  the  Taiwan  Security  Exchange  between 
the  years  1992-2002.  His  analysis  examines  situations  in  which  corporate  governance 
could  increase  firm  value. 

[Ting]  finds  positive  effects  of  corporate  governance  on  firm  performance, 
which  proves  the  existence  of  a  corporate  governance  effect.  Different  from 
the  previous  studies,  this  paper  addresses  the  fact  that  the  corporate 
governance  effect  exists  under  poor  economic  conditions.  As  expected,  firms 
with  poor  corporate  governance  mechanisms  tend  to  perform  badly  when 
business  cycles  go  downward.  In  other  words,  the  report  indicates  the 
importance  of  corporate  governance  increases  during  poor  economic 
conditions.  Firms  with  higher  agency  costs  also  show  a  significant  corporate 
governance  effect.  [Also,  when  investing  more  in  other  companies  causes  a 
firm  structure  to  become  more  complicated  while  simultaneously  diminishing 
information  disclosure,  the  corporate  governance  mechanism  could  work 
effectively.]  Finally,  the  recognition  of  a  supervisor  is  an  important  factor  for 
corporate  governance  effect  as  well.  Firms  with  executive  recognition  appear 
to  have  a  corporate  governance  effect.  (2006,  p.  8) 

To  summarize,  this  line  of  research  demonstrated  that  there  is  a  relationship 
between  corporate  governance  firm  values,  but  there  is  a  profound  gap  in  quantifying  the 
precise  value  of  the  management  that  governs  companies.  Furthermore,  once  the 
management/government  structure  is  broken  down  and  one  wishes  to  determine  the  value 
of  specific  managers,  the  lack  of  quantifiable  value  studies  becomes  even  more 
accentuated.  There  is  a  lack  of  studies  that  attempt  to  measure  the  value  of  management  in 
dollar  terms. 

Another  theme  of  corporate  management  valuation  studies  involves  discussing  the 
methods  of  increasing  the  value  of  corporate  management.  Unfortunately  for  our  purposes, 
this  literature  fails  to  produce  objective  measures  of  how  much  value  managers  add. 

Increasing  the  Value  of  Corporate  Management 

There  are  many  works  discussing  techniques  to  improve  management  techniques, 
and  most  offer  useful  insights  to  do  so.  However,  these  techniques  are  limited  because 
there  is  no  quantifiable  measure  to  discover  how  they  increase  firm  value. 

All  agree  that  more  effective  management  is  ideal,  but  seem  to  question  how  one 
can  discover  what  management  characteristics  and  techniques  are  the  most  effective 
without  discovering  the  relative  value  they  add.  How  can  managers  discover  what  they 
should  place  emphasis  on  without  knowing  the  payoff?  The  following  works  all  discuss 
valuable  techniques  crucial  to  manager  success,  but  none  supplement  their  advice  with 
techniques  to  discover  their  actual  value  added. 
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In  their  book,  Value  Driven  Management:  How  to  Create  and  Maximize  Value  over 
Time  for  Organizational  Success,  Pohiman  and  Gardiner  discuss  how  to  increase 
management’s  value  by  focusing  on  eight  “value  drivers”:  external  cultural  values,  internal 
cultural  values,  employee  values,  supplier  values,  customer  values,  third-party  values, 
competitor  values,  and  owner  values.  This  guideline  is  structured  to  help  managers  keep 
pace  with  fluctuating  business  structures  in  order  to  achieve  long-term  success.  Pohiman 
and  Gardiner’s  book  is  about  leading,  managing  and  working  in  organizations  as  managers 
enter  the  twenty-first  century.  Its  techniques  are  rooted  in  traditional  theories  and  styles,  but 
focus  on  value  creation  over  time — because  traditional  theories  must  constantly  make 
adjustments  as  paradigms  shift. 

Following  in  the  same  spirit  is  Michael  Armstrong’s  A  Handbook  of  Management 
Techniques  Revised,  3'^'^  edition.  Armstrong’s  work  acts  as  a  guide  for  professional 
managers  or  as  an  essential  guide  for  students.  His  work  attempts  to  encompass  value¬ 
adding  skills/techniques  for  numerous  management  responsibilities.  It  distinguishes 
between  tasks  that  fall  into  the  following  categories:  marketing  management,  operational 
management,  financial  management,  human  resource  management,  management  science, 
planning  and  resource  allocation,  efficiency  and  effectiveness.  Within  these  topics, 
Armstrong’s  work  includes  100  qualitative,  systematic,  and  analytical  methods  used  to  assist 
in  decision-making  and  to  improve  efficiency  and  effectiveness.  As  is  evident,  Pohiman  and 
Gardiner  as  well  as  Armstrong  focus  on  techniques  that  increase  management’s  value. 
Gardner’s  work  described  below  takes  a  slight  shift  in  topic  from  suggesting  useful 
techniques  to  highlighting  character  traits  that  are  inherent  in  a  successful  leader. 

John  Gardner’s  On  Leadership  (1990)  focuses  on  the  characteristics  a  leader  should 
possess,  centering  on  managers  and  how  such  qualities  increase  their  value  to  their 
business.  Gardner  lists  six  respects  with  which  leader/managers  distinguish  themselves 
from  the  general  run  of  managers: 

■  They  think  longer-term — beyond  the  day’s  crises,  beyond  the  quarterly  report, 
beyond  the  horizon. 

■  In  thinking  about  the  unit  they  are  heading,  they  grasp  its  relationship  to  larger 
realities — the  larger  organization  of  which  they  are  a  part,  conditions  external  to  the 
organization,  global  trends. 

■  They  reach  and  influence  constituents  beyond  their  jurisdictions,  beyond  boundaries. 

■  They  put  heavy  emphasis  on  the  intangibles  of  vision,  values,  and  motivation  and 
understand  intuitively  the  nonrational  and  unconscious  elements  in  leader- 
constituent  interaction. 

■  They  have  the  political  skill  to  cope  with  the  conflicting  requirements  of  multiple 
constituencies. 

■  They  think  in  terms  of  renewal.  The  routine  manager  tends  to  accept  organizational 
structure  and  process  as  it  exists.  The  leader  or  leader/manager  seeks  the  revisions 
of  process  and  structure  required  by  ever-changing  reality. 
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Management  characteristics  are  obviousiy  important  to  company  success. 

Therefore,  it  cieariy  foiiows  that  company  faiiure  can  resuit  from  choosing  a  manager  that 
iacks  the  characteristics  necessary  for  success.  Gerard  Egan,  in  Adding  Value:  A 
Systematic  Guide  to  Business-driven  Management  and  Leadership,  cites  this  as  one  of  the 
main  reasons  for  organizationai  faiiure.  He  provides  information  on  choosing  the  correct 
manager. 

When  companies  faii,  Egan  proposes  that  true  faiiure  iies  not  with  managers 
themseives,  but  with  the  system  by  which  they  are  chosen  and  deveioped.  Egan  states  that 
it  is  not  wise  to  promote  professionais  to  management  without  any  guidance  on  how  to 
actuaiiy  manage;  they  may  excei  in  a  specific  skiiis  area,  such  as  engineering  or  accounting, 
but  may  iack  the  specific  management  training  necessary  to  be  good  managers.  Egan 
describes  the  basic  skiiis  managers  need  to  iook  beyond  their  own  area  of  expertise  in  ways 
that  add  vaiue  to  the  business.  He  presents  a  comprehensive,  integrated  system  of 
management  that  can  be  adapted  to  meet  any  company's  reai  business  needs — inciuding 
strategy,  ieadership,  structure,  human  resources,  innovation,  and  organizationai  cuiture. 
Egan  offers  theoreticai  constructs  as  weii  as  three  practicai,  hands-on  modeis.  Most 
importantiy,  he  shows  how  to  integrate  the  modeis  into  a  system  that  managers  can  use  to 
identify,  organize,  and  impiement  the  best  ideas  emerging  from  today's  "business  and 
organizationai  potentiai"  movement. 

Aii  the  techniques  on  management  characteristics  discussed  in  countiess  books  may 
be  usefui,  but  they  do  not  address  the  issue  of  how  much  vaiue  such  techniques  and 
characteristics  can  add  in  specific  instances  with  specific  managers. 

The  Aipha  measure  in  fund  management  attempts  to  grasp  some  idea  of  the  vaiue  of 
management  in  generai,  but  aiso  has  severai  iimitations  that  make  it  probiematic  for  use  in 
determining  the  vaiue  added  of  individuai  managers. 

Lessons  from  Fund  Management  Research 

“Aipha”  measures  the  difference  between  a  fund's  actuai  returns  and  its  expected 
performance  given  its  ievei  of  risk  (as  measured  by  “beta”).  A  positive  aipha  figure  indicates 
the  fund  has  performed  better  than  its  beta  (risk)  wouid  predict,  in  contrast,  a  negative  aipha 
indicates  a  fund  has  underperformed,  given  the  expectations  estabiished  by  the  fund's  beta. 
Some  investors  see  aipha  as  a  measurement  of  the  vaiue  added  or  subtracted  by  a  fund's 
manager.  There  are  iimitations  to  aipha's  abiiity  to  accurateiy  depict  a  manager's  added  or 
subtracted  vaiue.  in  some  cases,  a  negative  aipha  can  resuit  from  the  expenses  that  are 
present  in  the  fund  figures  but  are  not  present  in  the  figures  of  the  comparison  index,  in 
addition,  aipha  is  dependent  on  the  accuracy  of  beta:  if  the  investor  accepts  beta  as  a 
conciusive  definition  of  risk,  a  positive  aipha  wouid  be  a  conciusive  indicator  of  good  fund 
performance.  Of  course,  the  vaiue  of  beta  is  dependent  on  R-squared. 

For  Aipha  vs.  the  Standard  index,  Morningstar  performs  its  caicuiations  using  the 
S&P  500  as  the  benchmark  index  for  equity  funds  and  the  Lehman  Brothers  Aggregate  as 
the  benchmark  index  for  bond  funds.  Morningstar  deducts  the  current  return  of  the  90-day  T- 
biii  from  the  totai  return  of  both  the  fund  and  the  benchmark  index.  The  difference  is  caiied 
the  fund's  excess  return.  The  exact  mathematicai  definition  of  aipha  that  Morningstar  uses  is 
iisted  beiow. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  - 180  - 


Alpha  =  Excess  Return  -  ((Beta  x  (Benchmark  -  Treasury)) 

Benchmark  =  Total  Return  of  Benchmark  Index 

Treasury  =  Return  on  Three-month  Treasury  Bill  (Morningstar). 

Aside  from  Alpha,  there  have  been  several  previous  attempts  to  quantify  the 
contributions  of  management,  but  they  have,  as  yet,  failed  to  provide  a  means  to  quantify 
the  actual  value  added  by  individual  managers.  More  recent  approaches  have  employed  a 
knowledge-based  metaphor  to  frame  the  problem. 

Knowledge-based  Approaches 

Housel  and  Nelson  (2005)  attempted  to  quantify  the  contributions  of  management  in 
aggregate  using  a  knowledge-based  framework.  The  general  idea  of  their  study  was  that  by 
quantifying  management’s  total  accumulated  education,  experience,  and  time  with  a  firm,  it 
was  possible  to  generate  its  weighting  on  the  output  of  a  firm. 

The  limitations  of  this  approach  were  that  it  did  not  provide  a  means  to  quantify 
individual  managers’  contributions;  it  assumed  that  management’s  aggregated  contributions 
were  purely  a  result  of  its  members’  combined  education,  experience,  and  time  with  the  firm, 
and  such  weightings  were  applied  as  a  constant  across  core  areas  or  processes  of  a  firm. 
While  useful  as  a  first  attempt  to  quantify  the  value  added  by  management  in  aggregate,  the 
general  approach  assumed  away  individual  differences  among  managers  in  terms  of  the 
value  they  add  to  a  firm;  likewise,  the  weighting  factor  could  not  be  directly  tied  to  the 
outputs  of  a  firm  in  a  relatively  unambiguous  way. 

Further,  it  would  be  quite  possible  for  two  firms  to  have  nearly  the  same  weighting 
factor  for  management  with  radically  different  profitability  and  productivity  scores.  A  more 
precise  methodology  that  can  differentiate  among  individual  managers  based  on  their 
observable  contributions  to  process  outputs  would  resolve  these  problems. 

Pavlou,  Housel,  Rodgers,  and  Jansen’s  (2005)  research  had  implications  for  the 
potential  value  added  by  managing  activities.  They  assumed  that  implicit  knowledge,  which 
is  akin  to  the  notion  of  the  type  of  knowledge  managers  use  in  making  creative  decisions 
(i.e.,  dark  matter  outputs),  could  be  accounted  for  in  terms  of  the  experience  of  employees 
(including  line  managers).  As  such,  a  simple  algorithm  to  measure  their  years  of  experience 
would  serve  as  an  indicator  of  the  amount  of  knowledge  used  to  produce  the  outputs  of 
creative  problem  solving.  However,  this  study  did  not  directly  address  the  issue  of  the  value 
added  directly  by  creative  managing  (i.e.,  management  dark  matter)  activities. 

There  is  a  substantial  collection  of  literature  regarding  corporate  governance, 
leadership  characteristics,  and  fund  management,  among  other  things,  which  attempts  to 
address  directly  or  indirectly  the  issue  of  how  to  measure  the  value  added  by  managing 
activities.  Yet,  there  remains  a  lack  of  research  that  attempts  to  objectively  quantify  the 
value  added  by  individual  managers.  Further,  past  approaches  do  not  provide  a  way  of 
structuring  the  problem  such  that  this  kind  of  objective  measure  can  be  derived  and  revenue 
or  value  can  be  allocated  to  individual  manager’s  dark  matter  outputs. 
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This  research  gap  is  further  accentuated  by  the  current  concerns  over  the 
transparency  of  corporate  activity,  which  assists  investors  in  making  more  informed 
decisions.  Transparency  would  aid  investors  in  understanding  the  rationale  for 
compensation  packages  provided  corporate  executives.  This  call  for  transparency  is 
particularly  important  for  businesses  in  the  United  States,  but  likely  applies  to  the  rest  of  the 
business  world  as  well. 

The  research  reviewed  leads  to  a  common  conclusion:  firm  governance,  leadership 
characteristics,  management  knowledge  and  experience  do  affect  firm  valuation.  Because 
firm  valuation  is  ultimately  a  result  of  a  firm’s  profitability  or  productivity  over  time,  there  is  a 
direct  relationship  between  firm  valuation  and  profitability  or  productivity  over  time.  The 
question  remaining  is,  “How  much  do  dark  matter  managing  activities  affect  firm 
profitability?” 

The  prior  research  also  provides  qualitative  recommendations  for  how  individual 
managers  can  increase  their  potential  value  to  the  firm.  However,  there  is  no  relatively 
objective  quantifiable  evidence  available  to  tie  given  characteristics  of  great  leaders  directly 
to  the  actual  profitability  or  productivity  of  a  firm. 

While  promising,  the  Alpha  measure  (the  term  sometimes  used  as  the  measurement 
of  value  a  manager  contributes  to  a  fund),  is  a  theoretical  measure,  is  difficult  to  estimate, 
and  is  seldom  reliable  because  it  is  very  difficult  to  operationalize.  Given  that  there  does  not 
appear  to  be  a  relatively  objective  way  to  quantify  the  value  added  by  individual  managers, 
such  a  measure  would  be  beneficial  to  both  managers  and  investors  alike. 

An  approach  to  estimating  how  much  value  managers  add  to  an  organization  or  fund 
would  provide  the  kinds  of  performance  data  that  might  be  used  to  reward  value-adding 
managers  while  not  rewarding  those  that  perform  poorly.  This  presumably  happens  today 
with  existing  performance-evaluation  techniques.  But  often,  these  techniques  appear  very 
subjective.  A  more  objective  technique  that  ties  performance  directly  to  the  firm’s 
profitability/productivity,  indicating  how  much  of  a  firm’s  revenue  can  be  allocated  to  given 
managers’  activities,  would  provide  a  more  convincing  evaluation. 

The  same  performance  information  might  lead  poorly  performing  managers  to  self- 
organize  in  a  way  that  ensures  they  are  in  positions  where  their  talents  can  be  used  in  the 
most  profitable,  productive  ways.  Such  measurement  would  also  provide  investors  with  the 
kind  of  performance  data  that  would  lead  them  to  reward  organizations  in  which  value  added 
by  management  benchmarks  well  within  an  industry  segment  while  withdrawing  support  for 
firms  in  which  management  does  not  perform  well. 

The  same  phenomena  should  occur  in  non-profit  organizations  as  well  when  relative 
productivity  among  organizations  can  be  compared  on  an  objective  basis.  The  federal 
government  with  its  stop-light  (Red=poor  performance,  Yellow=needs-improvement 
performance,  and  Green=good  performance)  scorecard  for  the  large  federal  agencies  is 
attempting  to  accomplish  this  goal.  However,  these  rather  crude  indicators  do  not  allow  for 
the  objective  quantification  of  agency  productivity,  let  alone  the  performance  of  agency 
management. 

What  is  needed  is  a  method  that  provides  a  way  to  quantify  individual  managers’ 
contributions  using  structural,  analytic,  and  relatively  objective  techniques  that  would  allow 
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comparisons  among  organizations.  The  method  we  proposed  above  promises  to  meet  these 
criteria  and  will  allow  allocation  of  revenue  to  managing  activities.  This  extension  of  the  KVA 
framework  allows  the  description  of  managing  activities  in  common  units.  In  addition, 
because  managers  produce,  interpret  and  send  messages  through  their  networks,  the 
method  also  accounts  for  these  managing  activities  in  a  common  descriptive  language. 

Options  and  Dark  Matter 

Many  management  decisions  are  constrained  by  legal  or  regulatory  frameworks  that 
severely  reduce  or  virtually  eliminate  management’s  ability  to  examine  alternative  future 
pathways  or  options.  If  there  are  no  options  for  managers  to  generate  and  from  which  to 
select,  there  is  no  purpose  for  dark  matter  activities. 

However,  where  options  exist,  managers  can  take  full  advantage  of  their  dark  matter 
capabilities  to  help  move  the  organization  toward  desired  future  states.  The  introduction  of 
open  business  acquisition  models  using  an  OA  system  development  framework  promise  to 
create  more  options  for  managers  to  achieve  their  objectives.  To  succeed  in  this  new 
environment,  managers  will  have  to  make  more  use  of  their  dark  matter  to  produce  the  kinds 
of  outputs  that  truly  serve  their  warfighting  customers.  Acquisition  managers  should  be  more 
free  and  motivated  to  use  their  dark  matter  capabilities  to  mitigate  potential  risks  while  taking 
advantage  of  upside  opportunities  to  build  better  systems  that  will  meet  warfighters’ 
changing  requirements  as  they  face  a  more  uncertain  environment. 

On  the  other  hand,  if  acquisition  managers  follow  management  practices  that  lead  to 
proprietary,  non-collaborative  solutions,  their  number  of  options  (and,  thus,  the  requirement 
for  the  use  of  their  dark  matter)  are  reduced.  This,  in  turn,  could  lead  to  a  reduction  in 
warfighting  capability  compared  to  situations  in  which  OA  and  open  business  models  are 
employed,  in  which  management  dark  matter  can  be  more  easily  utilized  by  acquisitions 
management. 

Framing  these  options  using  the  real  options  analysis  method  is  one  way  to  structure 
managers  analysis  that  also  takes  advantage  of  KVA  data.  This  technique  has  the  potential 
to  provide  managers  a  way  to  achieve  more  consistent  results  or  organizational  outcomes 
over  time.  Such  techniques  as  real  options  and  KVA  can  support  managers  in  producing 
more  informed  dark  matter  outputs  that  will  lead  to  better  outcomes  over  time. 

Benefits  of  Measuring  the  Vaiue  Added  of  Management 

This  method  for  measuring  the  performance  of  managing  activities  provides  a  variety 
of  new  kinds  of  information  for  executives,  investors,  and  managers.  These  include: 

■  a  method  to  test  the  value  of  different  management  techniques  (including  those 

advocated  in  prior  research) 

■  information  that  investors  could  use  to  determine  the  performance  of  managers 

■  new  ways  to  determine  a  salary  or  reward  system  based  on  managers’  verifiable 

contributions 
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a  basic  return  on  management  (ROM)  measure  at  any  level  of  aggregation  in  an 
organization 


■  performance  feedback  to  motivate  managers  to  best  utilize  their  individual 
talents/strengths 

This  new  information  would  allow  Adam  Smith’s  invisible  hand  of  the  competitive 
market  place  to  manifest  itself  in  determining  manager  rewards  and  would  allow  managers 
to  concentrate  on  techniques  that  are  proven  to  increase  their  value. 
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Appendix  A.  Future  Research — The  Delta  Problem 

The  ultimate  goal  of  the  approach  is  to  allocate  some  amount  revenue  to  amount  of 
management  dark  matter  of  outputs.  This  will  become  more  likely  when  we  have  the 
derivative  (coefficient)  necessary  to  convert  the  dark  matter  correlational  deltas  to  absolute 
numbers.  The  delta  will  be  proportional  to  revenue  with  a  given  coefficient  (that  is  yet  to  be 
discovered),  once  the  coefficient  is  established. 

Example  of  the  problem: 

Al  Smith,  manager  of  X  process,  generated  1  Gigabyte  of  (relevant)  dark  matter 
based  messages  during  January.  In  February,  he  generated  2  Gigabytes  of  dark  matter 
messages.  In  March,  he  generated  .5  Gigabytes  of  dark  messages.  In  January,  the 
company  made  $100  in  revenue,  in  February  it  made  $400  in  revenue,  and  in  March  it  made 
$200  in  revenue.  Correlating  Al’s  amount  of  dark  matter  messages  per  month  with  revenue 
per  month  we  establish  a  relatively  negative  correlation  between  the  two  values.  We  can 
then  check  the  volatility  of  the  company’s  revenue  performance  and  Al’s  dark  matter 
messages  over  the  same  3  months  and  correlate  these  volatilities.  Now  we  have  2 
correlations:  between  revenue  and  Al’s  amount  of  dark  matter  messages  and  between 
volatility  of  revenue  and  volatility  of  Al’s  amount  of  messages  per  month.  The  correlation 
coefficient  between  the  two  would  allow  us  to  eliminate  the  dark  messages  that  are  not 
related  to  the  revenue.  We  would  then  be  able  to  predict  the  manager’s  activities  based  on 
changes  in  revenue  or  vice  versa. 

We  need  to  establish  the  percentage  of  relevant  (i.e.,  dark  matter  based)  to  irrelevant 
manager  messages  for  every  time  period  in  terms  of  dark  matter  outputs.  The  volume  of 
irrelevant  messages  should  be  independent  of  the  prior  time  period.  The  amount  of 
irrelevant  messages  is  independent  from  revenue;  the  correlation  of  relevant  messages  to 
revenue  should  be  very  high  when  a  manager  is  influencing  corporate  outcomes.  The 
correlation  between  relevant  and  irrelevant  messages  also  should  be  very  low. 

These  basic  conditions  can  be  tested  in  empirical  research.  The  results  of  the 
research  should  help  us  establish  the  coefficient  that  will  allow  us  to  translate  correlational 
deltas  into  absolute  numbers  so  that  revenue  can  then  be  allocated  to  dark  matter  outputs. 

Appendix  B.  Correlating  the  Delta  in  Value  and  Management 
Dark  Matter  Activity 

Once  the  problems  of  establishing  a  method  of  bifurcating  relevant  and  irrelevant 
dark  matter  messages  has  been  resolved,  it  will  be  necessary  to  provide  the  method  for 
relating  changes  in  dark  matter  activity  with  changes  in  value  (e.g.,  revenue,  capability).  To 
do  this,  we  would  have  to  establish  a  baseline  dark  matter  measure  for  each  manager 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EORINEORMED  CHANGE  - 185  - 


against  which  to  calculate  the  rolling  averages  to  generate  the  delta  estimates.  The 
corresponding  time  periods  deltas  would  also  be  calculated  to  enable  the  correlations.  Over 
time  with  a  large  sample  size,  it  will  be  possible  to  estimate  the  optimal  number  of  dark 
matter  messages  for  a  given  level  of  environmental,  market  uncertainty. 

Assumptions  and  Algorithms 

In  what  follows,  we  lay  out  the  basic  algorithmic  framework  and  assumptions  for 
estimating  the  correlation  between  the  management  dark  matter  delta  and  value  (e.g., 
revenue,  capability)  delta. This  approach  will  assume  a  conservative  semantic 
interpretation  that  would  permit  estimation  of  amount  of  dark  matter  outputs  in  common 
units.  The  following  algorithms  are  a  preliminary  attempt  to  describe  the  delta  correlation 
approach. 

The  basic  equation  that  accounts  for  all  of  the  outputs  of  an  organization  at  a  given 
point  in  time  is: 

N  (i.e.,  number  of  firings  of  a  process,  activity)  x  A  (i.e.,  amount  of  complexity  for  one 

firing)  +  M  (i.e.,  relevant  dark  matter  management  activity).  Stated  more  simply: 

N  X  A  +  M  =  Total  Value  (T) 

To  measure  the  change  in  this  equation  from  time  period  1  to  time  2,  it  is  possible  to 
compute  the  total  value  produced  in  Ti  and  subtract  that  from  the  total  outputs  in  T2  in  the 
following  equation: 

Total  value  time  period  (Ti)  =  x  A^+ 

Total  value  time  period  (T2)  =  x  A^  + 

The  delta  for  value  over  the  two  time  periods  can  be  stated  as: 

(Ni  -  N2)  X  A  +  (Mi  -  M2)  =  delta  in  value  resulting  from  dark  management  activities. 

-  N''  (M^-  corresponds  to  ^  -  R'' 

(n^-N'')  X  a  +  (N^  -  N’)  corresponds  to  ^  -  N’ 

(Ma  -  ^  (Rd  -R.) 

A  +  (N4  -  N3)  ^  (N4  -  N3) 

M  -M^ _ ^  (R  -  R^)  _Rf_iR! 

N  -  (N  -  N^)  -  N' 

The  degree  of  change  from  one  period  to  the  next  resulting  from  this  dark 
management  activity  should  correspond  to  the  change  in  value  (e.g.,  revenue,  capability) 
from  the  same  two  time  periods.  This  equation  can  be  stated  as  follows  (where  V  =  revenue 
or  capability): 


The  approach  is  incomplete  at  this  juncture  because  we  do  not  have  the  coefficient  that  would  allow 
us  to  derive  an  absolute  number  (i.e.,  in  common  units  of  output)  that  would  lead  to  allocation  of  value 
to  management  dark  matter  activity. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EORINEORMED  CHANGE  - 186  - 


Ai(N2-Ni)/(Ni-N2)  xA  +  (M2-Mi)/(N2-Ni)^  (V2-Vi)/(Ni-N2) 


The  above  formulation  assumes  that  we  have  separated  out  irrelevant  messages 
from  M  and  that  M  represents  relevant  messages.  It  also  assumes  that  management 
messages  that  can  be  found  in  the  outputs  of  current  processes  are  algorithmically  definable 
and,  therefore,  accounted  for  using  the  routine  KVA  methodology.  This  formulation 
assumes  that  redundant  messages  have  been  eliminated  to  prevent  over-estimation  of  M. 
This  formulation  also  assumes  that  it  is  possible  to  derive  all  estimates  from  historical  data. 

Appendix  C.  Observations  about  Over-estimates  of  Dark  Matter 

Our  formulation  of  the  effect  of  management  dark  matter  activities  on  organizational 
value  can  lead  to  some  interesting  observations  about  managers  who  generate  dark  matter 
activities  that  may  not  contribute  to  organization  value.  For  example,  when  the  change  in 
value  is  0,  the  corresponding  change  in  M  should  also  be  0.  Managers  whose  generation  of 
dark  matter  messages  do  not  correlate  with  organizational  performance  may  be  creating  a 
lot  of  “churn”  but  little  value.  Given  a  large  number  of  time  periods,  the  manager  whose  dark 
matter  messages  do  not  correlate  with  organizational  performance  would  be  seen  as  one 
who  was  not  providing  unique  management  contributions  that  had  an  impact  on 
organization’s  value-generating  capabilities. 

This  formulation  does  not  reward  redundancy  in  management  dark  matter 
messages.  For  example,  the  manager  who  issues  the  command,  “Work  Harder!”  everyday 
for  a  given  time  period  would  only  get  credit  for  one  message  because  the  following  “Work 
Harder!”  messages  would  be  redundant  with  the  first.  Only  new  and  unique  messages  would 
be  counted  in  the  total  M  for  this  given  manager. 

This  formulation  also  would  lead  to  the  conclusion  that  management  dark  matter 
would  have  little  influence  on  organizational  value  generation  when  the  organization  was 
operating  in  a  very  stable  environment  with  little  risk  or  uncertainty.  There  should  be  a 
corresponding  increase  in  management  dark  matter  activity  when  an  organization 
encounters  turbulence,  risk,  and  even  higher  opportunities  for  increased  value.  It  follows  that 
the  complexity  of  a  management  environment  increases  in  correspondence  with  increases 
in  environmental  uncertainty  or  risk,  and  the  amount  of  dark  matter  messages  should  also 
increase  correspondingly  in  response.  For  example,  instructing  an  employee  to  “paint  the 
door  green”  in  routine  operations  is  much  less  complex  than  trying  to  predict  how  the  market 
will  respond  to  green  doors  as  tastes  change.  Similarly  in  the  military  environment,  an 
officer’s  instructions  to  move  supplies  from  point  A  to  point  B  in  peace-time  would  be  less 
complex  than  in  war-time,  when  there  are  increasing  risks  and  uncertainties  that  must  be 
dealt  with. 
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Wednesday, 
May  16,  2007 


1:45  p.m.-  chair: 
3:15  p.m. 


RDML  Kathleen  Dussault,  USN,  Deputy  Assistant  Secretary  of  the  Navy 
(Acquisition) 

Discussant: 

Jeffrey  P.  Parsons,  Director  of  Contracting,  Office  of  Command 
Contracting,  Headquarters,  US  Army  Materiel  Command 


Papers: 


Contractors  on  the  Battlefield:  When  and  How?  Using  the  US  Military's 
Risk  Management  Framework  to  Learn  from  the  Balkans  Support  Contract 

Victoria  A.  Greenfield,  US  Naval  Academy  and  Frank  Camm,  RAND 
Corporation 

Contractors  Supporting  Combat  Operations:  A  Failure  of  Imagination 
Richard  L.  Dunn,  University  of  Maryland 


Chair:  RDML  Kathleen  Dussault,  USN,  Deputy  Assistant  Secretary  of  the  Navy 
(Acquisition),  received  her  commission  from  Officer  Candidate  School  in  Newport,  Rhode  Island, 
in  November  1979.  She  graduated  from  the  University  of  Virginia  in  1 977  with  a  Bachelor  of  Arts 
in  American  Government.  Upon  graduation  from  Navy  Supply  Corps  School  in  May  1980,  she 
was  assigned  as  Assistant  Supply  Officer  and  Disbursing  Officer  to  the  Navy  Communications 
Station,  Nea  Makri,  Greece.  In  October  1981,  Rear  Adm.  Dussault  was  assigned  as  Supply 
Officer  in  USS  POINT  LOMA  (AGDS-2),  the  Pacific  Area  Launch  Support  Ship  for  the  Trident 
missile  program. 

In  June  1984,  Rear  Adm.  Dussault  was  assigned  to  Defense  Contract  Administration  Services 
Region  (DCASR),  Los  Angeles.  In  1986,  she  transitioned  to  duties  as  a  negotiator  and 
Contracting  Officer  at  Naval  Supply  Center,  Oakland,  California.  During  this  tour.  Rear  Adm. 
Dussault  completed  a  Master's  Degree  (with  honors)  in  Procurement  Management  from  Saint 
Mary's  College  in  Moraga,  California.  In  1990,  Rear  Adm.  Dussault  was  assigned  in  USS 
CONCORD  (AFS-5)  as  the  Assistant  Supply  Officer  where  she  performed  two  deployments 
contributing  to  the  logistics  success  of  Operations  Desert  Shield  and  Desert  Storm. 

In  October  1993,  Rear  Adml  Dussault  reported  to  Naval  Air  Systems  Command  (NAVAIR)  where 
she  performed  duties  as  Procuring  Contracting  Officer  for  the  Sidewinder  9X  new  development 
program.  In  1995,  she  served  as  the  Business  and  Financial  Manager  for  programs  managed  by 
the  Space  and  Naval  Warfare  Command.  In  June  1997,  Rear  Adm.  Dussault  served  as  the 
Executive  Assistant  to  the  Deputy  Assistant  Secretary  of  the  Navy  for  Acquisition  Management 
within  the  office  of  the  Assistant  Secretary  of  the  Navy  for  Research  Development  and 
Acquisition.  In  August  1998,  she  entered  the  Industrial  College  of  the  Armed  Forces  and  received 
her  Master's  Degree  in  National  Resource  Strategy  in  May  1999.  Rear  Adm.  Dussault  reported  as 
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Supply  Officer  of  USS  SEATTLE  (AOE-3)  in  August  1999  where  she  served  as  Afloat  Logistics 
Coordinator  while  deployed  to  the  Fifth  Fleet  operating  area.  In  May  2001,  she  assumed 
command  of  Defense  Distribution  Depot  San  Diego,  California.  In  April  2003  she  assumed 
command  of  the  Office  of  Special  Projects,  Arlington,  Virginia. 

Over  the  course  of  her  career.  Rear  Adm.  Dussault  has  achieved  the  highest  levels  of  certification 
in  Acquisition  and  Financial  Management  as  well  as  Joint  Professional  Military  Education.  She 
also  completed  the  Executive  Education  Program  at  Columbia  Business  School.  Her  decorations 
include  the  Defense  Superior  Service  Medal,  Legion  of  Merit,  Navy  Meritorious  Service  Medal 
with  two  gold  stars.  Joint  Service  Commendation  Medal,  Navy  Commendation  Medal,  Navy 
Achievement  Medal  with  gold  star,  and  various  unit  citations,  campaign  medals  and  service 
medals. 


Contractors  on  the  Battlefield:  When  and  How?  Using 
the  US  Military's  Risk  Management  Framework  to  Learn 
from  the  Balkans  Support  Contract^^ 


Presenter:  Victoria  A.  Greenfieid,  PhD,  holds  the  Crowe  Chair  in  the  Economics  of  the 
Defense  Industrial  Base  at  the  US  Naval  Academy  where  she  has  developed  an  upper-level 
undergraduate  course,  “The  Economics  of  the  Defense  Industrial  Base,”  and  leads  a  capstone 
research  seminar.  She  conducts  research  on  defense-industry  structure,  privatization  and 
outsourcing,  and  globalization:  international  narcotics  markets;  and  international  trade.  She  has 
published  recently  on  risk  management  in  military  contracting:  the  role  of  contractors  on  the 
battlefield:  technical  trade  barriers  facing  US  exporters;  defense  aerospace  globalization; 
narcotics  trafficking:  and  replacement  policies  for  aging  aircraft  under  uncertainty.  Dr. 
Greenfield’s  other  research  has  focused  on  strategic  planning  and  performance  measurement  for 
federal  agencies. 

Prior  to  joining  the  Academy,  Dr.  Greenfield  held  the  positions  of  Senior  Economist,  RAND 
Corporation;  Senior  Economist  for  International  Trade  and  Agriculture,  Council  of  Economic 
Advisers,  Executive  Office  of  the  President  of  the  United  States;  Chief  International  Economist, 
Bureau  of  Economic  and  Business  Affairs,  US  Department  of  State;  and  Principal  Analyst, 

Natural  Resources  and  Commerce  Division,  Congressional  Budget  Office.  Dr.  Greenfield  has 
advised  senior  US  policymakers  on  wide-ranging  domestic  and  international  issues,  including 
China’s  WTO  entry,  the  UN  Framework  Convention  on  Climate  Change  (member  of  US 
delegation),  the  US-Japan  civil  aviation  trade  agreement,  and  farm  policy  under  NAFTA. 


This  paper  draws  from  Greenfield,  V.A.,  &  Camm,  F.  (2005).  Performance  and  risk 
management  in  the  Balkans  Support  Contract  (MG-282-A).  Santa  Monica,  CA:  RAND,  which  can 
be  found  in  its  entirety  at  www.rand.ora/pubs/monoaraphs/MG282/:  a  related  presentation 
developed  by  Victoria  A.  Greenfield  for  “Contractors  on  the  battlefield:  Learning  from  the 
experience  in  Iraq,”  held  at  The  George  Washington  University  on  January  28,  2005,  and  a  paper 
presented  at  a  conference  of  the  American  Bar  Association  on  March  3,  2006.  We  thank  the 
sponsors  of  the  originating  RAND  report;  however,  we  bear  sole  responsibility  for  all  errors  or 
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Abstract 

For  centuries,  the  US  military  has  wrestled  with  decisions  about  when  and  how  to 
use  private  contractors,  especially  “Contractors  on  the  Battlefield.”  Reports  of  mixed 
performance,  inexperienced  contracting  officers,  miscommunication,  and  profiteering 
date  back  to  the  Revolutionary  War.  History  may  be  “living  history,”  in  part,  because 
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decision-makers  have  lacked  adequate  means  of  systematically  anticipating  future 
outcomes  and  harvesting  lessons  from  the  past.  The  US  military’s  risk-management 
framework,  a  familiar  tool  in  other  operational  settings,  may  fill  that  void.  To  illustrate, 
this  paper  applies  the  framework  to  the  Army’s  Balkans  Support  Contract  (BSC);  the 
contract  covers  a  variety  of  life  support,  transportation,  and  maintenance  services  and 
has  registered  a  substantial  track  record  in  deployment.  The  application  demonstrates 
the  utility  of  the  risk-management  framework  and  draws  general  lessons  from  the  BSC 
experience  for  selecting  service  providers  and  for  contract  development,  management, 
and  oversight.  Four  deceptively  simple  lessons  emerge  from  the  analysis:  first,  not  all 
risks  are  inherently  contractual;  most  are  environmental  or  activity-based.  Second,  risk  is 
dynamic;  appropriate  responses  change  over  time.  Third,  a  contract  is  only  as  good  as 
its  customer;  design  and  execution  determine  outcomes.  And  fourth,  risk  management  is 
not  risk  elimination;  not  all  risk  can  or  should  be  eliminated. 

Introduction 

For  centuries,  the  US  military  has  wrestled  with  decisions  about  when  and  how  to 
use  private  contractors,  especially  “Contractors  on  the  Battlefield.”  Under  what 
circumstance  should  it  hire  contractors  and,  if  it  does  hire  them,  what  can  it  do  to  insure 
that  it  gets  what  it  wants,  when  it  wants  it,  at  a  reasonable  cost? 

Shrader  (1999,  p.  3)  describes  the  use  of  contractors  in  the  Revolutionary  War 
and  the  apparent  timelessness  of  the  Army’s  experience: 

The  Army  experience  with  private  contractors  in  the  War  for  Independence 
contained  most  of  the  elements  which  would  characterize  the  later  use  of 
contractors  on  the  battlefield:  mixed  results  In  terms  of  performance  and 
adequate  support  for  the  troops;  lack  of  experience  and  expertise  on  the  part  of 
Army  officers  in  dealing  with  contractors;  lack  of  clarity  in  communications 
between  the  Army  and  supporting  contractors  as  to  requirements,  capabilities, 
and  costs;  and  financial  manipulation  and  desire  to  increase  profits  at  the 
expense  of  the  Army  on  the  part  of  contractors. 

Much  of  what  Shrader  describes  rings  true  today,  but  why?  History  may  be 
“living  history,”  in  part,  for  reasons  of  resources  and  culture,  e.g.,  the  number  of  billets 
allocated  to  contract  oversight  and  the  relative  status  of  contracting  officers,  and,  in  part, 
for  reasons  of  analytical  capability:  decision-makers  have  lacked  adequate — both 
tractable  and  broadly  applicable — means  of  systematically  anticipating  future  outcomes 
and  harvesting  lessons  from  the  past.  A  prospective  analysis  would  enable  planning, 
and  a  retrospective  analysis  would  enable  evaluation. 

The  US  military’s  risk-management  framework,  a  familiar  tool  in  other  operational 
settings,  offers  one  possible  means  of  filling  the  analytical  void:  it  is  simple  and  easy  to 
use  and  can  be  replicated  under  wide-ranging  circumstance.  The  framework  consists  of 
a  5-step  continuous  risk-management  process — (1)  identifying  hazards,  (2)  assessing 
hazards,  (3)  developing  controls  and  making  risk  decisions,  (4)  implementing  controls, 
and  (5)  supervising  and  reviewing — and  a  complementary  risk-assessment  matrix. 
Decision-makers  can  use  the  framework  to  recast  “when”  and  “how”  in  terms  more 
amenable  to  systematic  analysis:  what  risks  does  contracting  present;  how  do  they 
compare  to  the  risks  involved  in  using  US  military,  host  nation,  or  other  support;  and 
how,  if  at  all,  can  any  or  all  of  those  risks  be  mitigated — and  at  what  cost? 
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To  illustrate,  this  paper  applies  the  risk-management  framework  to  the  Army’s 
Balkans  Support  Contract  (BSC).  With  particular  attention  to  hazard  identification  and 
assessment,  the  application  demonstrates  the  utility  of  the  framework  and  draws  general 
lessons  from  the  BSC  experience  for  selecting  service  providers — of  different  types — 
and  for  contract  development,  management,  and  oversight. 

We  chose  the  BSC  because  it  covers  a  variety  of  life  support,  transportation,  and 
maintenance  services,  and  it  has  registered  a  substantial  track  record  in  deployment. 
Prior  to  the  war  in  Iraq,  it  was  also  the  largest  contract  of  its  kind,  both  in  terms  of  dollars 
and  contract  staff.  Among  its  limits  as  a  case  study,  the  BSC  does  not  cover  all  types  of 
services — it  does  not,  for  example,  provide  weapon  system  support. 

In  the  sections  that  follow,  we  review  the  origins,  principles,  and  structure  of  the 
BSC  as  the  basis  for  analysis,  present  and  apply  the  US  Military’s  risk-management 
framework,  and  summarize  our  findings. 

BSC  Origins,  Principles,  Structure,  and  Participants 

The  BSC,  as  described  in  this  paper,  began  operating  in  1999,  but  can  trace  its 
roots  to  the  first  Logistics  Civil  Augmentation  Plan  (LOGCAP)  contract,  which  the  Army 
awarded  to  Brown  and  Root  Services,  now  Kellogg  Brown  and  Root  (KBR),  in  1992.^® 
The  Army  established  LOGCAP  as  a  performance-based,  indefinite-delivery-indefinite- 
quantity  (IDIQ),  pre-planned  umbrella  contract,  capable  of  delivering  a  wide  range  of 
services  worldwide,  in  contingencies  or  crises,  on  demand.  Flexibility  and 
responsiveness  are  central  concepts  in  LOGCAP  and  other  such  contracts. 

■  A  performance-based  contract  tells  the  contractor  what  the  customer  wants 
done,  but  does  not  tell  the  contractor  how  to  do  it.  The  contractor  is  free  to 
leverage  its  resources,  including  expertise. 

■  An  IDIQ  contract  does  not  specify  the  delivery  date  or  exact  quantities  at  the  time 
of  the  award;  instead,  the  customer  orders  services  and  work  under  the  contract, 
as  needed.  This  level  of  generality  can  accommodate  uncertainty  about  timing 
and  quantities. 

■  In  a  pre-planned  contract,  the  contractor  develops  an  implementation  plan  for  a 
future  contingency,  ideally  in  close  collaboration  with  the  customer.  The  plan 
should  cover  all  activities  posited  in  the  statement  of  work,  potentially  reducing 
turnaround  times  on  service  and  work  orders. 


The  BSC  is  now  in  its  second  iteration;  the  first,  as  described  in  this  paper,  expired  in  2004.  On 
June  21 , 2005,  the  Army  awarded  a  re-specified  firm-fixed-price  and  cost-plus-award-fee  contract 
to  KBR  for  a  5-year  term;  the  Army  solicited  66  bids  on  September  29,  2003,  and  received  three. 
The  contract’s  spending  limit  is  $1 .25  billion.  (For  more  information,  see 
www.defenselink.mil/contracts  and  Halliburton  (2005)  at  www.halliburton.com.)  This  paper 
addresses  the  terms  of  the  BSC  as  established  in  1999  and  modified  over  the  primary  period  of 
research  encompassed  in  Greenfield  and  Camm  (2005),  i.e.,  2001-2003.  For  a  much  more 
detailed  treatment  of  the  contract,  see  Greenfield  and  Camm  (2005),  which  draws  material  from: 
CETAC  (1998),  CETAC  (1999),  CETAC  (2001),  and  CETAC  (2002);  CETAC-OC  (1997);  Kolar 
(1997);  McElroy  (1999);  USACE  (1994);  Wynn  (2000);  and  others.  For  a  particularly  readable 
overview  of  the  contract  and  the  roles  of  its  participants,  see  Wynn  (2000). 
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■  An  umbrella  contract  covers  a  broad  range  of  activities,  hence  the  term 

“umbrella,”  but  the  customer  can  choose  to  turn  to  other  service  providers  on  a 
task-by-task  basis  as  it  sees  fit;  indeed,  under  certain  circumstances  the 
customer  must  consider  using  other  providers,  including  other  contractors,  US 
military  personnel,  and  host-nation  support. 

The  Army  activated  the  LOGCAP  contract  in  the  Balkans  in  1995.  In  1997,  the 
Army  replaced  the  Balkans  element  of  the  LOGCAP  contract  with  a  sole-source  contract 
and,  almost  two  years  later,  the  Army  awarded  the  BSC  in  an  open  competitive  process 
based  on  “best  overall  value.”  The  selection  factors  for  the  BSC  award  consisted  of:  the 
management  execution  plan,  the  contractor’s  experience,  the  contractor’s  past 
performance,  and  cost  (CETAC,  1998).  The  cost  factor  spoke  to  realism,  completeness, 
and  financial  capability,  but  not  directly  to  level.  The  selection  process  weighed  all  non¬ 
cost  factors  equally — together,  they  were  deemed  “significantly  more  important”  than 
cost — and  considered  performance  risk  for  all  four  factors. 

The  principles  and  structure  of  the  BSC  closely  mirror  those  of  the  original 
LOGCAP  contract  and  impart  similar  flexibilities  to  both  the  customer  and  the  contractor. 
The  BSC  is  also  a  performance-based,  IDIQ,  pre-planned  umbrella  contract.  As  such, 
the  contract  confers  an  opportunity,  but  not  an  obligation  to  obtain  life  support, 
transportation,  and  maintenance  services.  The  performance-based  work  scope  tells  the 
contractor  what  needs  to  be  done,  e.g.,  that  the  Army  requires  laundry,  food,  waste 
removal,  road  repair,  or  other  services,  but  it  does  not  tell  the  contractor  how  many 
people  or  what  type  of  equipment  to  use. 

And,  as  was  true  of  the  LOGCAP  contract,  the  BSC  is  also  a  cost-plus-award-fee 
(CPAF)  contract. 

The  CPAF  payment  structure  warrants  further  consideration,  especially  with 
regard  to  the  incentives  it  creates  for  cost  control  and  cost  inflation.  The  bases  for 
award-fee  calculations,  cost  reimbursement,  and  future  contract  awards  each  play  a  part 
in  determining  those  incentives. 

First,  award-fee  payments  are  calculated  on  the  basis  of  negotiated  estimated 
costs,  not  actual  costs,  and  they  depend  on  the  results  of  regularly  scheduled 
performance  evaluations.  In  the  BSC,  the  contractor  may  obtain  an  award-fee  payment 
of  up  to  8%  of  the  negotiated  estimated  cost,  depending  on  its  performance  rating.  If,  for 
example,  the  contractor  receives  a  perfect  score  in  a  performance  evaluation  and  the 
estimated  negotiated  cost  of  the  service  it  provides  is  $100,  then  it  would  receive  an  $8 
award-fee  payment;  if  the  estimated  negotiate  cost  is  $50,  it  would  receive  a  $4  award- 
fee  payment;  and  so  on.  If  it  receives  a  less-than-perfect  score,  it  would  receive  a  pre¬ 
determined  fraction  of  the  $8  or  $4  award-fee  payment. 

If  actual  costs  differ  from  estimated  costs,  the  award-fee  payment  will  only 
change  if  the  difference  affects  the  contractor’s  performance  rating,  which  would 
depend,  in  turn,  on  the  specification  of  the  performance-evaluation  criteria.  If  cost  is 
among  the  criteria,  the  contractor  might  obtain  a  higher  rating  (and  payment)  for  lower 
costs  and  a  lower  rating  (and  payment)  for  higher  costs.  The  BSC  criteria  have  to 
varying  degrees  addressed  cost,  quality,  coordination,  flexibility,  and  responsiveness. 


Second,  CPAF  means  that  the  Army  reimburses  only  those  costs  that  are 
“reasonable,”  “allowable,”  and  “allocable”  under  the  contract  (Wynn,  2000,  p.  6).  The 
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Army  must  approve  the  expenditure.  If  the  contractor  spends  above  and  beyond  the 
cost  estimate,  the  contractor  may  not  get  that  approval.  Moreover,  even  if  the  contactor 
gets  the  approval,  it  may,  as  already  noted,  obtain  a  lower  rating  in  its  performance 
evaluation,  which  would,  in  turn,  result  in  a  lower  award-fee  payment. 

Third,  if  the  contractor  develops  a  reputation  for  inflating  costs;  that  is,  spending 
above  and  beyond  the  negotiated  estimated  costs,  the  Army  might  not  view  it  favorably 
in  future  competitions  for  other  contracts. 

Together,  these  three  considerations  suggest  that,  once  the  customer  and 
contractor  have  finished  negotiating  the  cost  estimate,  the  CPAF  structure  provides  little 
or  no  incentive  to  inflate  cost,  could  provide  incentive  to  control  cost,  and  may  even 
provide  incentive  to  reduce  costs,  if  reductions  result  in  higher  performance  ratings. 
Inflating  cost  will  not  generate  more  award  fee;  indeed,  it  may  result  in  less  fee  and 
fewer  future  contract  awards.  However,  tensions  may  arise  when  quality  is  also  a 
criterion,  particularly  if  additional  spending  can  improve  the  quality  of  service.^®  Higher 
costs,  all  else  constant,  might  result  in  a  lower  performance  rating,  but  higher  quality,  all 
else  constant,  might  result  in  a  higher  rating.  The  contractor’s  response  to  these 
competing  incentives  would  depend  on  their  reiative  weights  and  net  effect  in  the 
evaluation  process,  which  would  also  depend  on  the  preferences  of  the  customer. 

Ultimately,  the  CPAF  structure  places  the  burden  on  the  customer  to  clearly 
establish  and  articulate  its  needs  and  preferences  in  developing  the  contract,  to 
scrutinize  the  contractor’s  plans,  to  firmly  negotiate  appropriate  cost  estimates,  to  closely 
review  actual  costs  when  the  contractor  presents  them  for  approval  and  reimbursement, 
and  to  uphold  its  preferences  in  the  performance-evaluation  process.  The  analysis  of 
the  BSC  that  follows  confirms  each  of  these  points  in  part  or  whole. 

Finally,  the  contract’s  participants  constitute  a  near  “cast  of  thousands,”  spanning 
the  globe,  contributing  a  range  of  talents,  skills,  and  other  resources,  and  representing 
diverse  and  sometimes  conflicting  interests.  For  example,  US  Army  Europe  in  Germany 
is  the  bill-payer;  the  operational  units  deployed  in  the  theater  are  among  the  end-users; 
the  US  Army  Corp  of  Engineers  in  Winchester,  VA,  provides  the  Principle  Contracting 
Officer  (PCO);  and  the  contractor,  KBR  in  Houston,  TX,  provides  service.  The  Defense 
Contract  Management  Agency  and  the  Defense  Contract  Audit  Agency  also  provide 
substantial  administrative  support.  Notwithstanding  several  foregoing  references  to  “the 
customer”  as  if  it  were  a  single  monolithic  entity,  the  interests  of  the  bill-payer,  end- 
users,  PCO,  and  others  are  not  necessarily  the  same  and  at  times  may  be  at  odds  with 
one  another.  In  particular,  end-users  may  want  more  or  better  service  from  the  contract 
and  the  bill-payer  may  want  lower  cost,  leaving  the  PCO  to  address  resulting  frictions. 


^®  Economists  would  also  note  two  other  considerations:  first,  the  possibility  of  low-balling  in  the 
negotiation  to  win  the  award,  with  the  intent  to  charge  actual,  higher  costs  later;  and,  second,  the 
possibility  of  inflating  costs  in  the  current  period  to  increase  the  value  of  contracts  in  later  periods. 
In  the  first  instance,  issues  of  reimbursements  and  award-fee  determinations  could  discourage 
low-balling.  In  the  second  instance,  another  factor  would  come  into  play,  i.e.,  the  potential  for 
future  gain.  The  contractor  would  need  to  assess  the  effect  of  its  current  behavior  on  the 
likelihood  of  winning  a  future  contract  and  weigh  the  possible  sacrifice  of  current  earnings,  e.g., 
through  loss  of  reimbursement  or  award-fee  payment,  against  the  value  added  to  future 
contracts,  appropriately  discounted. 
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Given  the  PCO’s  “location”  between  the  end-users  and  the  bill-payer,  he  or  she  may 
have  a  strong  interest  in  minimizing  those  frictions. 

The  Risk-management  Framework 

Army  and  joint  doctrine  (Department  of  the  Army,  1998;  Department  of  the  Army, 
Marine  Corps,  Navy,  &  Air  Force,  2001)  provide  a  risk-management  framework  that  can 
be  used  to  identify  hazards,  assess  their  probability  and  severity,  establish  risk- 
mitigation  measures  or  risk  controls,  and  compare  risks  and  potential  responses  across 
alternative  service  providers.^®  The  framework  consists  of  a  5-step  continuous  risk- 
management  process  and  a  complementary  risk-assessment  matrix.  The  doctrine 
focuses  on  tactical  and  operational  considerations,  but  refers  to  other  applications, 
including  contracting.  Discussions  with  Army  and  other  military  personnel  suggest  that 
the  underlying  concepts  are  ingrained  in  their  thinking  about  deployment  generally,  but 
not  in  their  thinking  about  the  use  of  contractors  specifically. 

The  Army^^  defines  risk  as  the,  “chance  of  hazard  or  bad  consequences;  the 
probability  of  exposure  to  chance  of  injury  or  loss  from  a  hazard;  risk  level  is  expressed 
in  terms  of  hazard  probability  and  severity”  (Department  of  the  Army,  1998,  p.  Glossary- 
2).  The  Army  further  defines  hazard  as,  “a  condition  or  activity  with  potential  to  cause 
damage,  loss,  or  mission  degradation”  and  any  actual  or  potential  condition  that  can 
cause  injury,  illness,  or  death  of  personnel;  damage  to  or  loss  of  equipment  and 
property;  or  mission  degradation  (Department  of  the  Army,  1998,  pp.  Glossary-1  and  2- 
2).®^  In  loose,  non-technical  terms,  “bad  things”  can  happen  in  a  field  operation  or 
elsewhere,  with  varying  degrees  of  likelihood  and  impact. 

The  5-step  risk-management  process  unfolds  as  follows: 

First,  “Identify  hazards”  by  analyzing  the  mission,  listing  possible  hazards,  and 
listing  their  causes.  This  step  is  forward-looking:  for  example,  a  hypothetical  planner 
faces  concerns  about  a  service  provider’s  performance,  e.g.,  will  food  be  cold  or  roads 
impassable  (see  the  discussion  in  the  following  section).  The  planner  identifies  specific 
“bad  things”  and,  perhaps  as  importantly,  their  causes.  Failure  to  identify  the  causes — 
especially  the  sometimes  less  obvious  root  causes — could  lead  a  planner  to  suggest  the 
wrong  type  of  risk  control,  which  could  either  leave  the  risk  unchecked  or  create 
additional  risk.  An  assertion  that  “bad  things  can  happen”  won’t  suffice. 


“  Greenfield  and  Camm  (2005)  offer  a  more  detailed  treatment  of  risk,  the  risk-management 
framework,  and  the  applicability  of  the  framework. 

The  Army  (1998)  and  joint  (2001 )  doctrine  each  make  use  of  slightly  different  vocabulary  in 
their  discussions  of  risk;  this  analysis  draws  primarily  from  the  Army  doctrine. 

Department  of  the  Army  (1998,  p.  ii)  defines  “mission”  as  including  “mission,  operation,  or  task.” 
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Figure  1.  A  5-step  Continuous  Risk-management  Process 

(Department  of  the  Army  (1998);  Department  of  the  Army  et  al.  (2001 )) 


Second,  how  likely  is  it  that  the  “bad  things”  wiil  happen  and  how  bad  are  they? 
This  step,  assessing  hazard  probabiiity  and  severity,  can  be  used  to  estabiish  priorities 
for  allocating  resources  and,  speaking  to  the  issue  at  hand,  determining  whether  or  not  a 
contract  is  an  appropriate  vehicle.  As  shown  in  Figure  2,  Army  doctrine  provides  a 
simple  matrix  for  addressing  the  issues.  Third,  develop  controls  and  make  risk 
decisions.  With  a  reasonable  understanding  of  the  hazards  and  their  potential 
consequences,  the  planner  can  develop  controls,  determine  residual  risk,  and  make  risk 
decisions.  The  process  instructs  the  pianner  to  make  decisions  about  how  much  risk  to 
accept,  implying  that  it  may  not  be  possible  or  appropriate  to  eliminate  all  risk.  Fourth, 
implement  controls  in  the  operating  environment  and,  if  possibie,  before  entering  the 
operating  environment.  Fifth,  supervise  and  review  the  process  and  develop  new 
controls.  This  step  enables  continuous  feedback.  At  this  stage,  it  may  be  helpful  to  re¬ 
invoke  steps  one  and  two  to  review  any  realized  hazards — what  “bad  things”  have 
actuaiiy  happened,  what  caused  them,  and  how  bad  are  they? 
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Risk  Assessment  Matrix 

Probability 

Severity 

Frequent 

A 

Likely 

B 

Occasional 

C 

Seldom 

D 

Unlikely 

E 

Catastrophic 

I 

E 

E 

H 

H 

M 

Critical 

II 

E 

H 

H 

M 

L 

Marginal 

III 

H 

M 

M 

L 

L 

Negligible 

IV 

M 

L 

L 

L 

L 

E — Extremely  High  Risk 

H— High  Risk 

M — Moderate  Risk 

L — Low  Risk 

Figure  2.  Risk-assessment  Matrix:  How  Likely  and  How  Bad? 

(Department  of  the  Army  (1998);  Department  of  the  Army  et  al.  (2001 )) 


The  BSC  in  Theory  and  Practice 

This  section  focuses  on  applications  of  Steps  One  and  Two  of  the  5-step 
process:  first,  in  a  prospective  assessment  of  underlying  hazards  in  the  BSC;  second,  in 
a  retrospective  assessment  of  the  contract’s  performance. 

Underlying  Hazards 

Two  broad  categories  of  hazards  emerge  from  a  review  of  the  BSC  work  scope 
and  its  operating  environment.  One  category  relates  to  the  performance  and  cost  of 
day-to-day  activities,  such  as  food  preparation,  laundry,  road  repair,  and  waste  removal; 
the  other  to  higher-order  concerns  about  mission  success,  force  management,  and 
safety  and  security.®®  Context  matters.  Faulty  road  repairs  may  delay  a  postal  delivery 
in  one  instance  and  prevent  troops  from  reaching  the  battlefield  in  another. 

Framed  in  terms  of  the  risk-assessment  matrix  in  Figure  2,  a  planner  might  rank 
most  day-to-day  hazards  as  negligibly  or  marginally  severe  and  most  higher-order 
hazards  as  critically  or  catastrophically  severe. 

In  some  instances,  hazards  across  and  within  categories  are  interrelated.  For 
example,  seemingly  minor  day-to-day  problems  can  give  rise  to  higher-order  concerns 
overtime.  If  quality-of-life  suffers,  e.g.,  if  food  is  cold,  laundry  is  dirty,  and  latrines 
overflow,  the  Army  may  have  difficulty  recruiting  and  retaining  troops.  Moreover,  if  the 


®®  Mission  success  includes  readiness. 
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costs  of  these  services  are  too  high,^'*  the  Army  may  eventually  run  short  of  funding  for 
other  mission-essential  activities.  Hazards  may  also  involve  trade-offs  between 
competing  objectives,  especially  cost  and  quality.  Typically,  the  customer  must  pay 
more  to  get  more,  sometimes,  as  already  noted,  causing  tension  and  friction  among 
contractors,  bill-payers,  end-users,  and  PCO’s. 

Categorizing  and  recognizing  relationships  among  hazards  is  important  to 
understanding  the  nature  of  risk;  but  drilling  down  to  underlying  causality  yields  equally 
important  insight  for  developing  risk  controls. 

Consider  an  anecdotal  example:  a  soldier  returns  from  a  late  shift,  cannot  get  a 
hot  meal,  and  complains.  The  immediate  hazard  is  the  lack  of  a  hot  meal,  but  what  is 
the  cause?  The  most  obvious  answer  might  be,  “a  closed  kitchen,”  possibly  because  the 
contractor  is  short  staffed,  but  the  terms  of  the  BSC  point  elsewhere.  The  BSC  directs 
the  contractor  to  “Provide  24-hour  food  service  operations”;  however,  it  also  calls  for 
“limited  food  service  during  non-meal  hours”  (CETAC,  1998).  As  such,  the  contract  does 
not  require  hot  meals  after  the  late  shift.  It  permits  and  arguably  directs  the  opposite.  In 
this  case,  the  “cause”  Is  the  underlying  statement  of  work.  The  customer  is  getting  what 
it  asked  for,  nothing  more  and  nothing  less. 

Had  the  analysis  ended  with  a  closed  kitchen  or  short  staffing,  a  planner — or 
reviewer — might  have  suggested  the  wrong  control,  perhaps  a  staffing  mandate,  which 
might  have  increased  cost,  but  would  not  have  solved  the  problem. 

As  this  example  suggests,  one  possible  source  of  hazard  is  the  contract  itself  or, 
more  accurately,  a  poorly  framed  statement  of  work.  On  this  basis,  it  might  be  tempting 
to  argue  for  eliminating  the  contract — if  the  contract  is  the  source  of  the  hazard,  why  not 
eliminate  the  contract?  Merely  shifting  to  another  provider,  e.g.,  a  US  military  provider, 
will  not  resolve  the  problem  if  the  customer  continues  to  misstate  its  needs;  moreover, 
shifting  from  contract  personnel  to  troops  could  reduce  the  availability  of  troops  for  other 
activities  that  only  they  can  perform. 

Framing  can  be  made  better  or  worse  with  performance  standards.  In  a 
performance-based  contract,  such  standards  do  not  specify  “how”;  rather,  they  clarify 
“what”  and,  ideally,  provide  an  objective  basis  for  evaluation.  Without  standards,  the 
contractor  may  substitute  its  own  or  other  commonly  accepted  standards  and  may 
provide  the  wrong  service  or  the  wrong  amount  of  service.  However,  if  standards  are 
overly  prescriptive  they  may  negate  intended  flexibility. 

A  lack  of  performance  standards  may  also  contribute  to  so-called  gold-plating,  in 
which  a  contractor  provides  better  or  more  service  than  is  necessary  at  a  higher  cost. 

But,  gold-plating  cannot  occur  without  willing  participants  on  both  sides  of  the  contractual 
table.  A  CPAF  contract  might  create  incentives  to  over-provide  by  rewarding  higher- 
quality  service  with  a  higher  performance  rating,  but  the  customer  must  approve  the 
contractor’s  plan,  negotiate  the  cost  estimate,  and  authorize  reimbursements  for  actual 
costs.  Ultimately,  it  takes  two — or  more — to  gold-plate  effectively. 


^  Separate  from  concerns  about  intentional  cost  inflation,  reaiized  costs  might  be  higher  than 
expected  costs  within  the  confines  of  “reasonabie,  allowabie,  and  aliocable.” 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EORINEORMED  CHANGE  - 199  - 


other  possible  sources  of  hazard  include  ambiguous  roles  and  responsibilities 
and  inadequate  communication  and  coordination.  For  example,  a  contract  that  calls  for 
weekly  deliveries  of  fuel  oil  should  specify  whether  the  contractor  or  the  customer  is 
providing  the  fuel,  the  truck,  or  the  security.  If  the  customer  is  providing  any  of  these 
goods  or  services,  it  must  be  aware  of  and  plan  for  its  commitment;  moreover,  the 
customer  must  communicate  with  the  contractor  and  vice  versa,  e.g.,  regarding 
timetables  and  whereabouts,  to  coordinate  activities. 

More  recently,  attention  has  turned  to  two  other  issues  involving  “roles  and 
responsibilities,”  those  of  chain  of  command  and  protections  under  international  law. 
Contractors  operate  through  the  contracting  chain  of  command  and  outside  the  military 
chain  of  command,  which  has  raised  concerns  about  responsiveness  in  the  field;  in 
some  circumstances,  their  standing  under  international  law  is  uncertain,  which  has 
raised  concerns  about  their  security.^® 

BSC  Performance 

Realized  hazards  under  the  BSC  have  involved  day-to-day  activities,  with 
unsatisfactory  outcomes  arising  from  poorly  stated  service  or  work  orders,  inadequate 
communication  and  coordination,  and  incentives  to  accept  higher  costs  in  exchange  for 
higher  quality.  Higher-order  concerns  have  not  borne  fruit. 

A  GAO®®  report  (2000a)  on  controlling  costs  in  the  BSC  describes  an  instance  in 
which  planned  firefighting  services  would  have  been  too  costly  and  potentially 
unsupportable,  but  available  evidence  strongly  suggests  that  a  lack  of  performance 
standards  and  communication  between  the  customer  and  contractor  contributed  to  the 
conflict.  Concerns  about  unnecessary  redundancy  in  power  generation  also  seem  to 
have  had  their  roots,  at  least  partly,  in  standards  and  communication  issues.  The  GAO 
report  also  suggests  instances  of  gold-plating.  It  describes  spending  on  base  camp 
personalization,  e.g.,  changing  street  names  for  each  new  rotation  of  troops. 

Another  GAO  report  (2000b)  issued  at  about  the  same  time  finds  a  high  level  of 
customer  satisfaction  with  services  under  the  BSC.  The  concurrent  release  of  the  two 
GAO  reports — one  criticizing  spending  and  the  other  praising  quality-of-life — calls  to 
mind  the  inherent  tension  between  cost  and  quality.  The  two  reports  address  different 
costs  and  qualities,  but  the  juxtaposition  is  striking. 

Acknowledging  a  need  to  better  control  costs,  the  Army  responded  to  the  GAO 
by  clarifying  its  service  requirements,  by  publishing  a  handbook  of  performance 
standards,®’’  and  by  placing  greater  emphasis  on  cost,  especially  in  performance  criteria. 
It  re-oriented  the  criteria  to  weigh  cost  more  heavily  and  to  require  ongoing  reductions. 


®®  For  background  on  these  issues,  see  Department  of  the  Army  (2003);  for  a  more  general 
discussion  of  concerns  about  readiness  in  this  and  other  contracts,  see  GAO  (2003). 

®®  GAO  is  now  known  as  the  “US  Government  Accountability  Office”;  at  the  time  of  the  report  cited 
here,  it  was  known  as  the  “US  General  Accounting  Office.” 

®'^  The  Army  issued  a  call  to  more  closely  scrutinize  services  just  prior  to  the  publication  of  the 
GAO  report. 
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The  Army’s  response  may  reflect  a  natural  shift  in  priorities.  At  the  start  of  an 
operation,  the  Army  may  seek  to  allocate  managerial  talent  and  manpower  to  mission- 
essential  activities:  it  may  be  more  willing — and  able — to  devote  these  resources  to 
controlling  costs  as  the  operation  proceeds. 

Lessons  for  Future  Operations 

The  foregoing  analysis  applies  the  US  military’s  risk-management  framework  to 
the  BSC  to  consider  hazards,  both  prospectively  and  retrospectively.  The  results 
suggest  four  general  lessons  for  future  operations. 

First,  not  all  risks  are  inherently  contractual.  Few  of  the  hazards  uncovered  in 
the  BSC  were  “contractual”  per  se;  most  were  environmental  or  activity-based,  including 
those  stemming  from  trade-offs  that  the  Army — or  any  customer — routinely  faces, 
regardless  of  the  provider.  Were  the  hazards  attributed  solely  to  contracting,  any 
subsequent  risk  controls  (including  any  decisions  to  replace  contract  personnel  with 
troops)  might  be  ineffectual  or  worse.  Staffing  with  troops  will  not  guarantee  the  “right” 
services  if  the  customer  asks  for  the  “wrong”  services;  alternatively,  a  staffing  mandate 
will  not  correct  a  statement  of  work,  but  may  incur  unnecessary  costs. 

But,  risks  are  not  identical  across  providers.  The  probability  and  severity  of 
hazards  and  the  costs  of  controls  may  differ  by  type  of  provider,  as  in  the  case  of 
security  requirements.  Moreover,  in  some  instances,  the  hazards  themselves  will  differ, 
as  when  concerns  arise  about  the  chain  of  command  or  about  the  status  of  contractor 
employees  under  international  law.  Clearly,  the  results  of  the  analysis  will  be 
circumstance-specific  and  will  depending  on  conditions  in  the  operating  environment. 

The  real  lesson  is  to  compare  risks,  controls,  and  costs  across  potential  service 
providers — be  they  contractors,  US  military,  host  nation,  or  otherwise — considering  both 
common  and  unique  hazards  and  conditions  in  the  operating  environment.  For  example, 
a  contractor’s  employees  may  require  more  security  than  US  military  personnel, 
especially  in  an  unstable  operating  environment,  such  as  at  or  near  the  battlefield,  but 
the  Army  may  still  use  fewer  resources  on  balance  if  opts  for  the  contactor. 

Second,  risk  is  dynamic,  and  appropriate  responses  may  change  over  time. 
Prominent  concerns  at  the  start  of  an  operation  may  differ  vastly  from  those  at  the  end  of 
an  operation.  Action  typically  dominates  the  start.  Initially,  the  customer  must  focus  on 
getting  the  job  done  and  might  be  willing  to  pay  a  premium  to  do  so,  especially  if  it 
means  increasing  mission  effectiveness  by  freeing  up  managerial  talent  and  manpower 
for  other  purposes.  The  customer,  either  explicitly  or  implicitly,  may  be  considering  the 
“price”  of  controlling  cost  in  the  face  of  other  resource  constraints.  However,  as  the 
operation  proceeds  and  conditions  stabilize,  concerns  about  cost  tend  to  increase.  GAO 
criticisms  of  the  BSC  and  the  Army’s  response  suggest  that  cost  became  a  more  central 
theme  as  conditions  stabilized  in  the  Balkans. 
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Third,  the  contract  is  only  as  good  as  its  customer.  The  success  of  a  contract 
hinges  first  on  its  design  and  then  on  its  execution,  including  the  selection  process, 
management,  and  oversight.^®  This  means  that  the  customer  must  be  able  to: 

■  Clearly  establish  and  articulate  its  needs,  including  its  preferences  for  cost  and 
quality  in  the  face  of  tradeoffs; 

■  Carefully  review  the  contractor’s  plans,  firmly  negotiate  appropriate  cost 
estimates,  and  judge  the  validity  of  actual  costs; 

■  Uphold  its  preferences  during  the  performance-evaluation  process;  and 

■  Communicate  effectively,  coordinate  activities,  and  respond  to  changes  in  the 
operating  environment  as  they  arise. 

Here  too,  the  customer  faces  a  potential  hazard;  that  is,  if  it  is  overly  prescriptive 
in  stating  its  needs,  it  may  lose  some  of  the  flexibility  it  initially  sought  in  the  contract, 
especially  in  a  performance-based,  IDIQ  contract. 

A  participatory  “cast  of  thousands”  may  bring  a  richness  of  talents,  skills,  and 
other  resources  to  the  contract,  but  it  also  adds  complexity  and  potential  conflict  to 
design  and  execution.  By  implication,  each  participant — not  just  a  single  monolithic 
customer — must  understand  the  terms  of  the  contract  and,  as  already  noted,  its  roles 
and  responsibilities  under  the  contract.®® 

For  all  these  reasons,  good  training,  especially  with  potentially  less  familiar 
constructs  like  CPAF  contracting,  is  essential. 

Fourth,  risk  management  is  not  risk  elimination.  Not  all  risk  can  or  should  be 
eliminated,  but  decisions  about  how  much  risk  to  accept  should  be  made  consciously 
and  intentionally  with  adequate  information.  The  Army — or  any  other  customer — can 
structure  a  contract  to  address  many  forms  of  risk,  but  it  may  want  or  need  to  tolerate 
some.  The  costs  of  controls  might  be  too  high  in  view  of  the  probability  and  severity  of 
the  consequences.  Moreover,  as  conditions  in  the  operating  environment  change, 
decisions  about  risk  controls  and  acceptance  may  also  change. 
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Contractors  Supporting  Combat  Operations:  A  Faiiure  of 
Imagination 


Presenter:  Richard  L.  Dunn,  University  of  Maryland,  is  an  independent  consultant  and  Senior  Fellow 
at  the  University  of  Maryland.  He  conducts  research  in  national  security  operations  primarily 
emphasizing  the  analysis  of  laws,  policies,  and  practices  that  impact  the  effective  implementation  of 
technology.  His  research  includes  a  variety  of  acquisition  issues,  including  “contractors  on  the 
battlefield.”  He  has  previously  made  presentations  on  that  subject  at  the  NPS  Acquisition  Research 
Symposium  and  has  published  a  feature  comment  on  the  subject  in  Government  Contractor 
magazine. 

Dunn  retired  from  Government  after  serving  as  the  General  Counsel  of  the  Defense  Advanced 
Research  Projects  Agency.  Previously,  he  was  with  the  National  Aeronautics  and  Space 
Administration,  engaged  in  the  private  practice  of  law,  and  served  on  active  duty  as  a  Judge 
Advocate  in  the  USAF. 

Dunn  received  a  BA  (cum  laude)  from  the  University  of  New  Hampshire,  a  JD  from  the  University  of 
Maryland,  and  a  LLM  (Highest  Honors)  from  George  Washington  University. 


Abstract 

The  reasons  for  the  prevalence  of  contracted  support  for  military  deployments  and 
combat  operations  are  well  known.  In  the  1990’s,  the  military  force  structure  shrank  as  a 
dividend  of  the  end  of  the  Cold  War.  Contemporaneously,  outsourcing  proved  its  worth  in 
both  the  private  and  public  sectors.  The  Army  pioneered  a  logistic  civilian  augmentation 
program  (LOGCAP)  followed  by  somewhat  similar  programs  in  the  Navy  and  Air  Force. 
While  the  support  of  civilians  (both  government  and  contractor)  to  the  military  was  not  new, 
the  potential  scope  of  the  support  and  the  existence  of  on-call  world-wide  support  under 
LOGCAP  was.  In  addition,  the  sophistication  of  weapons  systems — especially  those  just 
entering  the  inventory — made  civilian-specialist  support  essential.  Moreover,  the  decision  to 
devote  more  of  the  shrinking  military  to  the  fighting  “tooth”  rather  to  than  the  supporting  “tail” 
tended  to  reduce  KP,  latrine  details,  and  similar  functions  that  had  once  been  hallmarks  of 
military  service. 

Currently,  the  public  debate  concerning  contractors  supporting  military  operations 
seems  to  be  dominated  by  “contracting”  concerns  rather  than  “military  operations”  concerns. 
Policy  is  driven  not  from  the  point  of  view  of  operations,  but  the  point  of  view  of  contracting 
with  its  myriad  rules  and  cast  of  players  (many  far  removed  from  the  scene  of  operations). 
Fortunately,  a  “one  team/one  fight”  spirit  seems  to  have  prevailed  so  far  in  combat  zones 
despite  questionable  bureaucratic  responses  and  examples  of  political  rhetoric  approaching 


demagogy. 


Iraq  dominates  the  current  debate  for  obvious  reasons.  It  is  the  scene  of  the  most 
widespread  and  intense  combat  operations  and  the  largest  contingent  of  contractors 
supporting  deployed  military  forces.  Iraq  is  also  the  prime  example  of  the  potential  mischief 
caused  by  a  policy  driven  by  the  contract  mentality  rather  than  the  operations  mentality.  No 
one  is  in  charge!  No  one  person,  no  one  commander  is  in  charge  of,  or  even  knows  how 
many,  contractors  are  in  Iraq.  This  is  not  only  a  management  problem  but  a  potential  legal 
(and  perhaps  not  too  much  of  a  stretch  to  say,  moral)  problem. 
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While  Iraq  serves  as  an  example  of  the  frailty  of  current  policy,  the  real  failing  is  a 
lack  of  vision  or  imagination.  The  lessons  of  history  and  examples  from  other  countries 
should  be  considered  in  crafting  the  optimum  policy  for  contracted  support  for  military 
operations.  World  War  II  is  rich  in  lessons  for  various  responses  to  problems  we  confront 
today;  many  of  those  lessons  have  been  ignored.  Foreign  countries  have  crafted  policies, 
such  as  Britain’s  sponsored  reserves,  which  while  not  panaceas,  could  prove  beneficial  If 
intelligently  adapted  and  implemented  in  the  environment  of  the  American  military. 

Our  current  system  fails  on  a  number  of  scores.  Its  failure  of  imagination  is  also  a 
failure  to  implement  some  obvious  and  tried  principles.  The  first  principle  that  is  ignored  is 
unity  of  command.  This  should  apply  to  the  contractors  who  are  now  so  important  to  the 
success  of  military  operations  as  well  as  to  the  military.  Part  of  the  problem  is  that  our  so- 
called  “contracting”  system  is  more  a  regulatory  system  than  true  contracting.  A  contracting 
regime  should  have  as  its  prime  principle  facilitating  transactions  rather  than  regulating 
them.  Personnel  in  a  combat  zone  should  be  subject  to  certain  common  standards.  A 
common  code  governing  criminal  conduct  is  prime  among  these.  This  research  concludes 
that  while  policy  development  has  addressed  many  important  issues,  it  has  lacked  vision 
and  failed  to  address  or  even  recognize  key  top-level  issues. 
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Panel  7  -  Complexity  and  Collaboration  in  Acquisition 
Organizations 


Wednesday, 
May  16,  2007 


1:45  p.m.-  Chair: 
3:15  p.m. 


Dr.  Robert  Buhrkuhl,  Director,  Joint  Rapid  Acquisition  Cell,  Office  of  the 
Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics) 


Discussant: 


Lt.  Col.  Lawrence  S.  Ryder,  USMC,  Deputy  Program  Manager,  Joint  High 
Speed  Vessel,  PEO  Ships 


Papers: 


Application  of  a  Network  Perspective  to  DoD  Weapon  System  Acquisition: 
An  Exploratory  Study 

Nancy  Roberts  and  Ryan  Mantz,  Naval  Postgraduate  School 

Developing  Collaborative  Capacity:  A  Diagnostic  Model 

Susan  Page  Hocevar,  Erik  Jansen  and  Gail  Fann  Thomas,  Naval 
Postgraduate  School 


Chair:  Dr.  Robert  Buhrkuhl,  Director,  Joint  Rapid  Acquisition  Cell,  Office  of  the  Under  Secretary  of 
Defense  (Acquisition,  Technology  and  Logistics),  was  born  in  Slater,  Missouri  and  graduated  from 
Central  Missouri  State  University  in  1969,  with  a  Bachelor  of  Science  degree  in  Law  Enforcement  and 
Political  Science.  He  earned  a  Master  of  Business  Administration  degree  from  that  same  institution  in 
1971,  and  was  simultaneously  designated  a  Distinguished  Military  Graduate  and  commissioned  a 
Second  Lieutenant  in  the  United  States  Army.  Dr.  Buhrkuhl  continued  his  formal  civilian  education  by 
earning  a  Doctor  of  Philosophy  degree  in  Public  Policy  Analysis  and  Administration,  St.  Louis 
University,  in  1981 .  His  professional  military  and  civilian  education  also  includes:  Military  Police 
Officer  Basic  Course;  Civil  Affairs  Officer  Advanced  Course;  Psychological  Operations  Course; 
Defense  Resource  Management  Course;  Planning,  Programming,  and  Budgeting  Course;  Command 
and  General  Staff  College;  Army  War  College;  Program  Management  Course;  Senior  Executive 
Leadership  Course;  and  the  Defense  Leadership  and  Management  Program,  Class  of  1997. 

Dr.  Buhrkuhl  is  a  Level  III  Certified  Acquisition  Professional  in  Program  Management  and  Business 
Financial  Management.  He  is  the  author/co-author  of  five  publications,  two  regarding  Zero  Based 
Budgeting,  one  regarding  Transportation,  one  regarding  the  implementation  of  the  Program  Executive 
Officer  (PEO)  concept  in  Army  Aviation,  and  one  regarding  Joint  Warfighting  Experiments.  Dr. 
Buhrkuhl  was  also  directly  involved  in  the  "standing-up"  of  the  Comanche  Program  Management 
Office  (PMO),  the  Special  Operations  Aviation  PMO,  the  Combat  Support  Aviation  PEO,  and  the 
Comanche  Business  Office  at  Texas  Instruments. 

In  his  current  position.  Dr.  Buhrkuhl  is  responsible  for  leading  the  JRAC,  which  consists  of  a  core 
group  of  Senior  Executive  Service  and  General  Officer  personnel,  in  meeting  urgent  materiel  and 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  -  207  - 


logistics  requirements  which  have  been  certified  as  being  operationally  critical  by  the  Combatant 
Commanders,  the  military  services,  and  the  Defense  agencies. 

In  his  prior  AT&L  position.  Dr.  Buhrkuhl  provided  ongoing  leadership  and  acquisition  management 
oversight  for  82  Major  Defense  Acquisition  Programs  (MDAPs)  and  pre-MDAPs  from  the  Army,  Navy 
and  Air  Force  whose  total  acquisition  funding  exceeds  $1,117  trillion.  Additionally,  he  was  responsible 
for  all  monthly  Defense  Acquisition  Executive  Summary  briefings  to  the  USD(AT&L):  Earned  Value 
Management  reporting;  submission  of  Selected  Acquisition  Reports  to  Congress;  and  the  review, 
processing,  and  coordination  of  all  MDAP  Acquisition  Strategies,  Acquisition  Program  Baselines,  and 
Acquisition  Decision  Memorandums  that  were  the  result  of  Defense  Acquisition  Board  deliberations. 

Prior  to  assuming  his  position  at  the  USD(AT&L),  Dr.  Buhrkuhl  was  the  Director  of  Management, 
Special  Operations  Acquisition  and  Logistics  Center,  United  States  Special  Operations  Command, 
MacDill  AFB,  where  he  was  responsible  for  Program  Integration  (including  Life  Cycle  Cost  Estimating 
and  PPBS);  Acquisition  Policy  and  Logistics  (including  Foreign  Comparative  Testing  );  Information 
Technology  Management;  Management  Operations;  and  the  Technical  Industrial  Liaison  Office.  His 
past  assignments  also  include:  Acting  Deputy/Assistant  PEO,  Aviation  PEO;  Program  Analysis 
Officer,  Aviation  PEO  and  Combat  Support  Aviation  PEO;  Budget  Officer,  Troop  Support  Command; 
and  Supervisory  Budget  Analyst  and  Program  Analyst,  Aviation  Research  and  Development 
Command.  Dr.  Buhrkuhl  has  also  worked  in  private  industry  on  two  occasions.  In  1985,  he  worked  as 
the  Business  Manager,  Helicopter  Systems  Branch,  Texas  Instruments,  Inc.;  and  from  October  1989 
to  March  1991,  he  worked  as  a  Senior  Program  Manager,  DCS  Corporation. 

Dr.  Buhrkuhl  also  managed  a  second  career  as  an  officer  in  the  United  States  Army  Reserve.  As  an 
Army  reservist  for  over  23  years,  he  attained  the  rank  of  Colonel  and  served  in  a  number  of 
leadership  positions  including:  Commander;  Budget  Officer;  Team  Leader;  and  Management  Analyst. 
For  the  last  eight  years  of  his  Army  Reserve  career,  he  was  assigned  to  the  Office  of  the  Assistant 
Secretary  of  the  Army  for  Research,  Development  and  Acquisition,  Pentagon. 

Dr.  Buhrkuhl's  civilian  awards  include:  Exceptional  Civilian  Service  Award;  Meritorious  Civilian 
Service  Award;  Superior  Civilian  Service  Award;  six  Special  Act  Awards;  and  numerous  Exceptional 
Performance  Awards  and  Letters  of  Commendation.  His  military  awards  and  decorations  include: 
National  Defense  Service  Ribbon;  Meritorious  Service  Medal;  three  Army  Commendation  Medals; 
three  Army  Achievement  Medals;  and  the  Armed  Forces  Reserve  Medal. 

Dr.  Buhrkuhl  is  married  to  the  former  Bonnie  L.  Bruce  of  Maplewood,  Missouri.  They  have  one  son. 
Brad,  who  lives  in  Atlanta,  Georgia  and  is  an  Information  Management  Technology  Specialist. 
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Application  of  a  Network  Perspective  to  DoD  Weapon 
System  Acquisition:  An  Expioratory  Study 


Author:  Nancy  Roberts  is  a  Professor  of  Defense  Analysis  in  the  Graduate  School  of  Operational 
and  Information  Sciences  at  the  Naval  Postgraduate  School  in  Monterey,  California.  She  received  a 
PhD  from  Stanford  University,  a  MA  and  BA  from  the  University  of  Illinois,  and  a  Diplome  Annuel, 
from  the  Cours  de  Civilization  Francaise  at  the  Sorbonne.  Her  previous  faculty  appointments  have 
been  at  the  Graduate  School  of  Business  at  the  Naval  Postgraduate  School,  the  Carlson  School  of 
Management  at  the  University  of  Minnesota,  and  the  Graduate  School  of  Business  at  Stanford 
University  as  a  visiting  associate  professor.  Dr.  Roberts  has  published  extensively  in  the  areas  of 
public  entrepreneurship  and  innovation,  strategic  management  and  planning,  leadership,  stakeholder 
collaboration,  complex  networks,  dialogue  and  deliberation.  Her  recent  work  focuses  on  “wicked 
problems”  such  as  the  organizational  challenges  of  peace  operations  and  post-conflict  reconstruction. 
She  is  the  co-author  of  Transforming  Public  Policy:  Dynamics  of  Public  Entrepreneurship  and 
Innovation  (1996J  and  editor  of  two  books — The  Transformative  Power  of  Dialogue  (2002)  and  Direct 
Citizen  Participation  (2007). 

Dr.  Roberts  is  also  a  co-editor  of  a  book  series  on  Research  on  Public  Management  for  Information 
Age  Publishing,  an  Associate  Editor  of  PAR,  and  serves  on  the  editorial  boards  of  Public 
Management,  The  American  Review  of  Public  Administration,  International  Public  Management 
Journal,  and  International  Public  Management  Review.  She  currently  teaches  courses  on  Tracking 
and  Disrupting  Dark  Networks,  Planning  and  Organizing  in  Complex  Networks,  and  Coping  with 
Wicked  Problems. 

Presenter:  Major  Ryan  Mantz,  USAF,  has  served  in  various  program  management  and  acquisition 
staff  positions  for  over  14  years.  He  is  currently  serving  as  the  Chief  of  Systems  Engineering  at  the 
NATO  Airborne  Warning  and  Control  Programme  Management  Agency  in  Brunssum,  the 
Netherlands.  As  the  Lead  System  Integrator  in  the  Implementation  Division,  Maj  Mantz  oversees  the 
technical  performance  of  upgrades  to  improve  the  effectiveness  of  the  NATO  AWACS  fleet  and 
training  systems.  Major  Mantz  has  also  served  at  both  Air  Staff-  and  Center-level  positions,  including 
policy/transformation  action  officer.  Program  Manager,  and  Deputy  Program  Manager.  He  was 
Program  Manager  for  the  development  projects  on  the  Air  Force  Control  Network  where  his 
communication  upgrades  enabled  satellite  command  and  control  over  commercial  communication 
links.  He  has  also  served  as  Deputy  PM  for  the  ACAT  I,  ICBM  Propulsion  Replacement  Program. 

Major  Mantz'  education  includes  a  Master's  of  Business  Administration  from  the  Naval  Postgraduate 
School,  a  Master's  of  Science  in  Environmental  Science  from  the  University  of  Texas  at  San  Antonio, 
and  a  Bachelor's  of  Science  in  Economics  from  the  United  States  Air  Force  Academy. 


Abstract 

One  of  the  foundations  of  military  command  and  control  is  that  authority  must  match 
responsibility.  Yet  in  weapon  system  acquisition,  a  program  manager  is  responsible  to 
deliver  capabilities  to  the  warfighter  without  full  control  of  the  resources  he  needs  to  carry 
out  this  task.  Successful  program  managers  recognize  their  dependencies  upon  other 
actors  and  execute  their  programs  using  a  network  with  a  common  goal  of  enhancing  a 
specific  warfighting  capability.  A  hierarchical  chain  of  command  still  exists,  but  the  network 
enables  the  actors  to  carry  out  their  objectives  in  an  efficient  and  effective  manner.  This 
report  describes  how  the  acquisition  process  purportedly  works  in  hierarchical  terms.  It  also 
introduces  a  process  model  to  describe  the  set  of  activities  actually  used  and  the  actors  who 
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are  required  to  collaborate  to  deliver  capabilities  to  the  warfighter.  The  analysis  of  those 
activities  between  actors  reveals  that  weapon  system  acquisition  behaves  like  a  network. 
Describing  acquisition  in  network  terms  allows  those  involved  in  weapon  system  acquisition 
oversight,  policy,  and  practice  to  have  new  insights  and  measurement  tools  to  understand 
how  to  improve  the  weapon  systems  acquisition  process. 

Introduction 


Prelude 


Over  one-hundred  years  ago,  the  Wright  Brothers  were  the  first  to  accomplish  a 
manned,  controlled,  heavier-than-air-flight,  making  history  at  Kitty  Hawk,  North  Carolina,  on 
December  17,  1903.  How  did  two  bicycle  mechanics  from  Dayton,  Ohio,  accomplish  this 
feat  against  a  host  of  inventors?  And,  why  did  the  Wright's  lose  their  advantage  and  not 
continue  to  make  aviation  history?  The  answer  to  both  questions  revolves  around  their 
networks.  Early  on,  the  Wright's  were  not  only  inventors,  they  were  networked  innovators. 
Shulman  concluded  that  their  early  success  was  due  to  their  correspondence  and  sharing  of 
ideas  with  Samuel  Langley  and  flight  historian  Octave  Chanute,  who  had  built  an  extensive 
network  within  the  aviation  community  (2002).  Following  their  successful  flight,  however,  the 
Wright's  network  was  limited  through  secrecy  that  was  driven  by  a  desire  to  patent  the 
airplane  and  secure  a  monopoly,  even  Chanute's  request  for  information  about  their  maiden 
flights  (Shulman,  2002).  The  Wrights  cut  themselves  off  from  their  network,  preferring  to 
secure  the  patents  rather  than  build  upon  their  technological  feat.  The  loss  of  their  network 
also  led  to  stagnation  in  their  innovation  efforts.  Glenn  Curtiss,  on  the  other  hand,  was 
anything  but  secretive.  He  and  the  Aerial  Experiment  Association  built  his  June  Bug  aircraft 
and  demonstrated  flying  to  the  public.  Eventually,  Curtiss'  collaborative  network  yielded  the 
invention  of  500  aviation  devices,  many  of  which  are  still  in  use  today.  His  factory  invented 
and  sold  the  flying  boat  to  the  Navy,  along  with  6,000  JN-1  Jenny's,  making  Curtiss  Aircraft 
one  of  the  largest  aircraft  companies  in  the  world  (Shulman,  2002).  In  essence,  the  duel 
between  the  Wrights  and  Curtiss  proved  that  the  success  of  complex  projects  Is  predicated 
upon  the  structure  of  the  project's  network  of  collaborators. 

Would  Curtiss  recognize  today's  billion-dollar  weapon  system  programs  with  their 
high-stakes  decision-making  process  ensuring  that  entrepreneurs  do  not  waste  precious 
taxpayer  resources?  Or,  has  the  world  not  changed  that  much...  Do  successful  programs 
still  collaborate  and  network  to  successfully  deliver  capabilities  to  warfighters? 

Acquisition  Process  Problems 

Department  of  Defense  (DoD)  weapon  system  acquisition  programs  are  plagued  with 
performance  shortfalls,  and  even  more  notably,  cost  and  schedule  overruns.  Addressing 
this  problem  has  spawned  numerous  studies  and  reforms  over  many  years.  Most  recently, 
the  push  to  reinvent  government  in  the  1990s  resulted  in  a  series  of  reforms  that  led 
acquisition  toward  a  market-based  model.  Despite  these  efforts  to  improve  efficiency, 
success  has  yet  to  be  realized,  with  several  recent  studies  noting  increasing  cost  and 
schedule  overruns.  Civilian  and  military  officials  at  the  highest  levels  In  the  Pentagon  have 
expressed  frustration  at  the  lack  of  balance  among  the  competing  interests  of  cost, 
schedule,  and  performance  in  weapon  system  acquisition  programs.  Given  many 
stakeholders  with  multiple  interests  in  the  acquisition  process  and  the  inability  of  high- 
ranking  officials  to  achieve  a  balance  among  competing  interests,  assigning  a  program 
manager  responsibility  for  balancing  cost,  schedule,  and  performance  appears  to  be  a 
nearly  impossible  task. 
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In  addition  to  problems  managing  costs,  schedule,  and  performance,  warfighters  are 
asking  even  more  from  their  weapon  systems,  requiring  capabilities  that  are  joint, 
interoperable,  and  able  to  seamlessly  share  information.  Joint  staffs  are  looking  to  gain  an 
advantage  on  the  battlefield  based  upon  a  revolution  in  military  affairs  driven  by  the 
explosion  in  information  technology.  A  weapon  system  program  manager  must  manage  not 
only  her  own  baseline  but,  in  addition,  rely  on  capabilities  from  other  systems  that  are  also  in 
development. 

Alternative  Acquisition  Organizations 

Along  with  many  initiatives  to  solve  the  fundamental  acquisition  problem,  the 
strategic  assumptions  underlying  acquisition  reforms  point  to  three  alternative  organizations: 
hierarchical  control,  market  solutions,  or  network  collaboration.  Powell  (1990)  concluded 
that  hierarchies,  markets,  and  networks  are  the  three  basic  forms  of  organization. 
Congressional  and  politically  appointed  civilian  control  of  the  weapon  system  acquisition 
process,  along  with  the  chain  of  command  within  the  DoD,  makes  one  think  of  acquisition  in 
hierarchical  terms.  Alternatively,  weapon  system  acquisition  relies  heavily  on  contractors 
who  possess  the  know-how  and  resources  to  produce  major  weapon  systems.  A  market- 
based  solution  to  acquisition  problems  is  also  rational.  Finally,  acquisition  programs  create 
the  need  to  cross  organizational  boundaries  for  decision-making — necessitating  the  need  for 
a  network  form  of  governance. 

The  policy-makers  and  practitioners  within  the  weapon  system  acquisition  process 
do  not  typically  think  of  the  process  in  network  terms.  Yet,  Powell  (1990)  concluded  that 
networks  are  the  predominant  form  of  organization  with  a  very  few  pure  markets  or 
hierarchies  in  existence.  This  project  is  devoted  to  describing  the  acquisition  process  in 
network  terms.  Therefore,  the  research  question  for  this  paper  is:  Does  the  DoD  weapon 
system  acquisition  process  behave  as  a  network? 

The  focus  of  this  project  is  to  understand  how  weapon  system  acquisition  programs 
accomplish  their  objectives,  and  whether  those  interactions  fit  within  the  description  of  a 
network.  This  analysis  will  offer  a  new  perspective  on  the  acquisition  process. 

Methodology 

Chapter  II  describes  the  acquisition  process  and  its  interactions  with  both  the 
warfighters  who  describe  weapon  system  capability  needs  and  the  budget  staff  who  balance 
alternative  needs  against  fiscal  constraints.  A  process  model  will  be  introduced  to  describe 
the  full  set  of  activities  and  interactions  a  program  must  go  through  from  concept  to  delivery 
and  operation. 

With  the  activities  of  the  acquisition  process  in  mind.  Chapter  III  highlights  the 
characteristics  of  networks.  A  definition  of  networks  is  established,  and  aspects  of  networks 
are  described  from  a  review  of  literature.  Several  network  analysis  techniques  are  coupled 
with  a  description  of  operating  within  networks,  allowing  an  analysis  of  the  acquisition 
process  in  network  terms  in  Chapter  IV. 

Finally,  Chapter  V  offers  conclusions  to  the  basic  research  question  of  whether 
weapon  system  acquisition  may  be  described  in  network  terms.  Further,  several 
recommendations  are  offered  to  improve  this  analysis  and  further  apply  a  network  model  to 
acquisition. 
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Weapon  System  Acquisition  Process 

The  Department  of  Defense  (DoD)  weapon  system  acquisition  process  must  be 
described  before  it  can  be  characterized  as  a  hierarchy,  network,  or  market.  This  Chapter 
will  describe  the  acquisition  process  and  its  interactions  with  other  key  processes.  To 
analyze  these  interactions,  a  detailed  process  model  will  be  introduced  that  describes  the 
activities  and  actors  involved  in  transforming  inputs  into  outputs  in  the  form  of  knowledge 
and,  ultimately,  weapon  systems. 

Background 

The  mission  of  defense  acquisition  is  to  deliver  needed  capabilities  to  warfighters.  In 
the  hands  of  warfighters,  these  capabilities  are  able  to  produce  constructive  effects  on  the 
battlefield.  The  defense  acquisition  system  is,  in  essence,  developing  the  set  of  equipment 
that  will  be  used  to  fight  the  next  war.  The  process  of  collaboration  among  competing 
agencies  to  make  these  decisions  is  a  very  complex  task  that  combines  optimization  of 
doctrine,  organization,  training,  materiel,  leadership  and  education,  personnel,  and  facilities 
(DOTMLPF)  solutions  within  the  Joint  Capabilities  and  Integration  Development  System 
(JCIDS).  Additionally,  these  decisions  are  dynamic,  changing  over  time  in  response  to 
environmental  variables.  This  results  in  changing  desires  and  continuing  debate  over  what 
is  the  best  solution. 

Further,  delivering  materiel  capability  requires  a  complex  set  of  actors,  and  even 
more  stakeholders,  who,  from  markedly  different  perspectives,  seek  to  optimize  the  various 
processes  of  technology  development,  integration,  test  and  evaluation,  production,  fielding, 
and  sustainment  of  weapon  systems.  Nevertheless,  the  governing  directive  within  the  DoD, 
Directive  5000.1,  gives  the  Program  Manager  the  purported  authority  and  the  clear 
responsibility  to  deliver  required  capabilities  to  the  warfighter  (Department  of  Defense, 
2003a).  Therefore,  the  Program  Manager  must  find  ways  to  shape  the  capability  needs 
from  the  JCIDS  requirements  generation  system;  choose  a  design  architecture,  mature 
technologies,  and  develop  an  acquisition  strategy  within  the  Defense  Acquisition  System; 
and  seek  resources  from  the  Planning,  Programming,  Budgeting,  and  Execution  (PPBE) 
System.  Dynamic  interaction  among  these  systems  is  required  to  deliver  a  capability  to  the 
warfighter.  Kadish,  et  al.,  described  this  interaction  as  the  "Big  A"  acquisition  process 
(2006).  This  paper  will  use  this  cross-cutting  definition  of  the  acquisition  process. 

This  chapter  will  highlight  the  key  processes  and  interactions  required  to  deliver  a 
capability.  The  JCIDS,  Defense  Acquisition  System,  and  PPBE  system  will  be  briefly 
examined.  A  process  model  will  be  introduced  to  highlight  the  depth  and  complexity  of  the 
interactions  the  acquisition  process  must  perform  to  deliver  a  capability. 

Joint  Capabilities  Integration  and  Development  System  (JCIDS) 

The  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  was  born  out  of 
the  perception  that  each  service  parochially  examined  alternatives  within  its  own  core 
competencies,  rather  than  from  the  perspective  of  a  joint  warfighting  environment.  The 
Goidwater-Nichols  Act  of  1 986  created  a  framework  where  Combatant  Commanders 
(COCOMs)  are  responsible  for  joint  operations,  and  service  secretaries  and  commanders 
are  responsible  to  organize,  train,  and  equip  the  military  to  conduct  army,  naval,  and  air 
operations  in  support  of  the  combatant  commanders  {Goidwater-Nichois  Act,  1986).  The 
Goidwater-Nichols  Act  gave  the  COCOMs  a  significant  voice  in  the  funding  process.  JCIDS 
essentially  took  the  next  step  and  institutionalized  a  process  in  which  requirements  are 
jointly  conceived,  validated,  and  approved  prior  to  each  service  implementing  those  needs. 
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The  other  effect  of  JCIDS  is  to  define  capabilities  gaps  rather  than  threat-driven 
needs.  The  Chairman,  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3170.01  E  defined 
capabilities  as; 

The  ability  to  achieve  a  desired  effect  under  specified  standards  and 
conditions  through  combinations  of  means  and  ways  to  perform  a  set  of 
tasks.  It  is  defined  by  an  operational  user  and  expressed  in  broad  operational 
terms  in  the  format  of  a  joint  or  initial  capabilities  document  or  a  joint  doctrine, 
organization,  training,  materiel,  leadership  and  education,  personnel,  and 
facilities  (DOTMLPF)  change  recommendation  (Chairman  of  the  Joint  Chiefs 
of  Staff,  2005). 

JCIDS  Pattern  of  Relationships 

The  drivers  of  the  JCIDS  process  are  the  representatives  of  the  warfighting 
community.  The  Combatant  Commands  and  Joint  Staff  run  key  portions  of  the  process. 

The  services’  requirements  communities  become  involved  as  they  sponsor  an  approach  that 
falls  inside  one  of  their  warfighting  core  competencies.  One  difficulty  in  the  JCIDS  process 
is  getting  the  services  involved  without  corrupting  the  process  by  making  it  a  forum  for  the 
each  service  to  argue  for  its  preferred  approach.  JCIDS  is  supposed  to  avoid  this  problem 
through  Joint  Staff  analysis  of  capability  gaps  identified  by  the  Combatant  Commands. 

Several  presentations  at  the  PEO/SYSCOM  Conference  in  December  2003  outlined 
what  are  essentially  opposing  views  on  the  service's  role  during  a  panel  on  aligning  JCIDS 
and  the  Defense  Acquisition  System.  Dr  Glenn  Lamartin,  OSD(AT&L)  Director  of  Defense 
Systems  noted  throughout  his  briefing  that  the  new  JCIDS  and  Acquisition  policies  had  to  be 
followed  with  collaborative  relationships  between  the  OSD,  the  Functional  Capabilities 
Boards,  and  the  Services  to  support  decision-making  (2003).  Dr.  Nancy  Spruill,  OSD(AT&L) 
Director  of  Acquisition  Resources  and  Analysis,  supported  a  view  that  the  OSD  ought  to  be 
the  decision-maker  in  the  process,  holding  cross-cutting  Defense  Acquisition  Boards  and 
either  cutting  or  accelerating  service  programs  to  meet  joint  needs  (2003).  Essentially,  Dr 
Spruill  viewed  the  services  as  materiel  providers  who  would  react  to  OSD-defined  solutions, 
whereas  Dr  Lamartin  valued  the  services’  inputs  to  the  joint  architectures  and  decisions  as  a 
critical  interdependency.  The  right  viewpoint  is  the  one  that  recognizes  how  information  is 
distributed.  If  information  that  is  needed  for  decision-making  is  distributed  within  the 
services  and  the  combatant  commands,  the  services  ought  to  be  involved.  If  the  Combatant 
Commands  and  Joint  Staff  have  the  information  they  need  to  derive  alternatives  that 
integrate  with  current  warfighting  systems  and  doctrine,  then  the  services  might  be  viewed 
as  implementers  of  systems. 

JCIDS  Realities 

As  structured  as  the  JCIDS  process  appears,  the  reality  is  that  requirements  change 
over  time.  As  technological  possibilities  and  threat  conditions  change,  so  do  needs  of  the 
warfighter.  Within  the  acquisition  system,  "requirements  creep"  may  show  up  late  in  the 
process  in  the  form  of  expectations  or  actual  changes  to  written  requirements.  JCIDS 
institutionalized  this  concept  with  the  Capabilities  Production  Document,  offering  the 
opportunity  for  requirements  changes  before  entering  low-rate  initial  production  (Matthews, 
2004).  Further,  the  expectations  of  the  warfighter  are  often  not  met  in  a  timely  manner 
because  expectations  evolve  over  time.  Without  changing  written  requirements,  the 
operational  community  may  interpret  what  was  previously  stated  in  a  requirements 
document  differently.  Therefore,  both  the  perception  and  the  reality  is  that  the  desired 
outputs  of  the  acquisition  system  are  dynamic. 
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Planning,  Programming,  Budgeting,  and  Execution  System 

The  funding  for  the  program  comes  through  the  PPBE  process.  Every  other  year, 
the  OSD  issues  budget  guidance,  and  the  services  begin  a  biannual  cycle  of  preparing 
program  objective  memorandums  (POM)  to  advocate  their  program's  needs  among  other 
service  priorities.  Eventually,  the  OSD  comptroller  and  the  Office  of  Management  and 
Budget  (0MB)  prepare  the  defense  portion  of  the  President's  Budget.  Even  though 
Congress  normally  appropriates  money  for  only  each  fiscal  year,  the  POM  for  a  program 
portrays  the  budget  reflected  in  the  Future  Year  Defense  Program.  This,  in  essence,  gives 
the  budget  community  a  forecast  of  what  the  budget  will  look  like  to  satisfy  spending 
priorities  for  the  next  several  fiscal  years. 

The  Planning,  Programming,  Budgeting,  and  Execution  system  is  a  centralized, 
structured  way  of  allocating  resources  to  support  the  National  Security  Strategy.  McCaffrey 
and  Jones  described  the  goal  of  PPBE  as  balancing  forces,  equipment,  and  support  given 
resource  constraints  (2004).  Given  the  competitive  nature  of  the  services,  this  process 
allows  the  Secretary  of  Defense  to  balance  competing  objectives  and  select  the  most 
beneficial  use  of  resources. 

The  overlap  of  the  planning,  programming,  budgeting,  and  execution  phases,  along 
with  the  multitude  of  disparate  stakeholders,  makes  the  system  very  complex.  Nonetheless, 
there  is  structure  from  the  strategies  of  the  planning  phase,  to  the  alternatives  of  the 
programming  phase,  the  constraining  of  the  budgeting  phase,  and  finally,  the  execution 
phase  where  funds  are  appropriated,  allocated,  re-allocated,  and  expended.  Lewis,  Brown, 
and  Roll  contend  that  the  Air  Force  budget  process  includes  centralized  planning  and 
decentralized  execution  with  the  Major  Commands  (MAJCOMs)  playing  a  key  role  as  the 
interface  with  the  COCOMs.  The  JCIDS  process  generates  capability  needs  that  flow 
through  Air  Staff  to  OSD  to  become  part  of  the  budget.  Budget  and  manpower  flow  through 
Air  Staff  to  the  program  office  for  execution  (2002). 

Defense  Acquisition  System 

The  Defense  Acquisition  System  refines  concepts;  matures  technologies;  develops 
and  integrates  system  designs;  and  tests,  produces,  sustains,  and  disposes  of  weapon 
systems  in  response  to  warfighter  needs.  The  Department  of  Defense  Directive  {DODD 
5000.1)  (Department  of  Defense,  2003a,  sec.  3.2)  governing  weapon  system  acquisition 
defines  an  acquisition  program  as:  "a  directed,  funded  effort  that  provides  a  new,  improved, 
or  continuing  materiel,  weapon  or  information  system  or  service  capability  in  response  to  an 
approved  need." 

The  sponsor  (i.e.,  a  Major  Command  in  the  Air  Force)  uses  the  JCIDS  process  as 
outlined  above  to  define  the  need.  The  interface  with  the  acquisition  community  is  through 
the  Initial  Capabilities  Document.  This  input  is  refined  in  the  concept-refinement  phase 
through  the  Analysis  of  Alternatives  process  to  select  a  materiel  alternative  that  Is 
operationally  and  cost-effective.  The  sponsor  is  responsible  for  the  analysis  of  alternatives 
using  a  collaborative  process  with  the  acquirer,  developer,  tester,  and  other  enabling 
communities  to  refine  the  "art  of  the  possible"  (Air  Force,  2005,  p.  9). 

The  acquisition  process  uses  a  high-level  framework  as  shown  in  Figure  1  that 
serves  as  a  common  reference  and  set  of  expectations  for  all  programs.  The  reality  is  that 
every  program  is  unique.  An  infamous  retort  within  the  acquisition  community  when  asked  a 
general  question  about  acquisition  programs  is,  "It  depends."  The  expectations  for  each 
program  are  established  through  the  milestone  decision  authority  at  a  milestone  decision. 
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Figure  1.  Acquisition  Phases  and  Milestones 

(Department  of  Defense,  2003b) 
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Despite  many  interdependencies  across  the  acquisition  stakeholder  community,  DoD 
Directive  5000.1  names  the  milestone  decision  authority  and  program  manager  as  key 
participants.  The  milestone  decision  authority  is  given  overall  responsibility  for  the  program, 
while  the  program  manager  is,  "the  designated  individual  with  the  responsibility  for  and 
authority  to  meet  program  objectives"  (2003a).  The  reality,  however,  is  that  the  program 
manager  must  collaborate  among  many  interests  to  accomplish  program  objectives. 
Collaboration  using  integrated  product  teams  (IPT)  is  the  tool  designated  to  resolve 
competing  interests.  The  collaborative  process  is  not  specified  in  detail,  although  DoD 
Directive  5000.1  (2003a)  lists  the  communities  that  ought  to  participate  in  collaborative 
decision-making  and  identifies  the  IPT  as  the  entry  point  for  organizations  that  want  to 
collaborate.  The  program  manager  and  milestone  decision  authority  use  the  IPTs'  advice  to 
make  better  decisions  (Department  of  Defense,  2003a). 

Weapon  system  acquisition  process  model 

Purpose 

Given  a  plethora  of  the  stakeholders  and  a  complex  product-development  process, 
the  set  of  interactions  required  to  manage  a  program  need  to  be  well  understood. 

Describing  the  process  to  manage  an  acquisition  program  helps  assess  who  interacts  and 
how  they  interact  to  accomplish  a  program.  The  Assistant  Secretary  of  the  Air  Force 
(Acquisition  Integration),  SAF/AQX,  formed  the  Acquisition  Process  Action  Team  (APAT)  in 
Spring  2005  to  describe  the  set  of  processes  Air  Force  weapon  systems  were  using  to 
accomplish  their  missions.  The  goals  were  to  baseline  the  acquisition  processes  and  form  a 
common  language  and  basis  of  measurement  across  the  stakeholders  in  the  acquisition 
process.  The  group  focused  mainly  on  the  defense  acquisition  system  itself  and  its 
interactions  with  JCIDS  and  PPBE. 

Lt.  Col.  Michael  Paul  and  Major  Ryan  Mantz,  SAF/AQXA,  led  the  APAT  effort.  A 
group  of  consultants  from  the  Center  for  Reengineering  and  Enabling  Technologies  (CRET) 
provided  the  methodology  and  manpower  to  support  the  data-gathering  effort.  Mr.  Mike 
Wilhelm,  CRET,  was  instrumental  in  managing  the  effort. 

In  order  to  assess  the  interactions  within  weapon  system  acquisition,  the  APAT  used 
an  enterprise  process-model  approach.  A  process  model  offers  a  look  across  the  many 
disciplines  within  weapon  system  acquisition  to  understand  what  behaviors  the  team  is 
using  to  solve  the  problem.  The  model  is  put  into  process  terms,  where  each  step  is  defined 
as  a  verb-subject  relationship.  Instead  of  describing  a  contracting/source  selection  process. 
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the  step  is  simply  "Select  Source."  This  allows  the  team  to  focus  on  the  stakeholders'  Inputs 
to  the  process  instead  of  driving  the  description  solely  in  contracting  terms. 

Another  important  aspect  of  a  process  model  is  to  describe  the  relationship  between 
the  steps  and  other  actors.  In  essence,  the  process  model  is  a  look  at  the 
interdependencies  within  the  acquisition  system.  Each  step  in  the  process  is  described  in 
terms  of  inputs,  outputs,  triggers,  and  mechanisms.  A  source  of  those  characteristics  is  also 
described.  This  allows  the  model  to  describe  interaction  with  other  steps  in  the  process. 

Data  Gathering 

The  APAT  team  used  the  DOD  5000  series  regulations  as  a  jumping-off  point.  The 
major  steps  in  the  process  were  chosen  as  the  high-level  steps  in  the  process.  This  allowed 
the  model  to  refer  back  to  a  reference  to  which  acquisition,  logistics,  finance,  contracting, 
test,  and  requirements  personnel  could  relate.  Beginning  with  the  high-level  process,  the 
APAT  team  held  several  workshops  with  a  core  group  to  decompose  the  high-level  process 
into  a  series  of  lower-level  process  steps.  To  ensure  that  the  process  model  reflected  the 
interactions  across  the  Air  Force  acquisition  process,  the  team  established  a  series  of 
workshops  with  acquisition  personnel  to  refine  the  second-level  of  the  model  and  develop 
the  third  and  lower-levels  of  the  model.  Each  workshop  lasted  approximately  two  days  and 
was  focused  on  a  particular  phase  of  the  acquisition  process.  Participants  from  all  bases 
were  invited,  but  the  main,  working-level  participants  were  from  the  host  base.  A  series  of 
workshops  were  held  at  the  Pentagon,  Eglin  AFB,  Warner-Robins  AFB,  and  Wright- 
Patterson  AFB,  which  gathered  120  collective  participants  from  across  acquisition  functions 
of  requirements,  engineering,  test,  program  management,  finance,  sustainment, 
maintenance,  and  disposal.  Further,  telephone  conferences  were  held  to  refine  the  results. 

Results 

The  team  used  the  following  definitions  as  part  of  the  process-decomposition 

effort: 


Process 

Inputs 

Outputs 

Suppliers 

Customers 

Key  Players 

Controls 

Mechanisms 


Logical  set  of  steps  transforming  an  input  into  an  output 

Information  or  resource  consumed  in  the  activity  to  create  the  output 

Information  produced  by  an  activity 

Those  who  provide  the  input  to  the  process 

Those  who  receive  the  output  of  the  process 

Those  ultimately  responsible  for  the  process  being  accomplished 

Business  rules  that  govern  the  performance  of  an  activity 

Resource  that  performs  or  supports  an  activity,  but  is  not  consumed 
by  the  activity 


Processes  were  decomposed  into  roughly  five  to  seven  sub-processes  that  were  the 
key  components  of  the  higher-level  process.  The  workshop  participants  were  instructed  to 
keep  decomposing  processes  until  they  were  defined  at  an  "actionable  level."  In  reality,  the 
processes  were  decomposed  until  workshop  participants  could  not  agree  on  sub-processes 
that  generally  fit  most  programs. 


Appendix  A  depicts  the  output  from  the  APAT  effort.  The  APAT  effort  identified  27 
process  steps  supporting  the  five  major  DoD  5000  acquisition  phases.  Beneath  the  major 
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processes  are  107  sub-processes  with  172  supporting  activities.  The  workshop  participants 
were  more  comfortable  with  the  latter  three  phases  of  the  acquisition  process  than  the  first 
two.  Concept  Refinement  and  Technology  Development  had  fewer  sub-process  and 
supporting  activity  steps  upon  which  participants  were  able  to  agree. 

Even  more  important  than  the  numbers  of  steps  are  the  key  players,  suppliers,  and 
customers  of  each  process  step.  To  make  the  data  more  manageable  for  this  paper,  key 
sub-  processes  and  supporting  activities  were  chosen  in  the  Concept-refinement, 
Technology-development,  and  System-design  and  Development  phases  of  the  acquisition 
process.  These  phases  shape  the  program  and  lock-in  the  design  characteristics  that  affect 
cost  schedule  and  performance  during  the  latter  phases.  Therefore,  this  paper  focuses  on 
these  early  phases  of  acquisition. 

What  is  a  Network? 

Introduction 

Chapter  II  defines  both  how  weapon  system  acquisition  purportedly  and  actually 
behaves.  There  is  a  defined,  hierarchical  chain  of  command  with  a  milestone  decision 
authority  and  a  program  manager  who  is  responsible  for  delivering  a  weapon  system 
capability.  The  APAT  process  study  also  revealed  that  the  inputs  required  to  deliver  this 
capability  require  a  set  of  stakeholder  interactions  that  go  outside  the  boundaries  of  the 
traditional  chain  of  command.  Further,  the  stakeholders  involved  have  differing  and 
dynamic  objectives  causing  both  real  and  perceived  instability  within  the  acquisition  process. 
First,  however,  to  address  the  question  of  whether  the  defense  acquisition  system  can  be 
characterized  as  a  network,  one  must  first  define  networks  and  understand  their  basic 
properties. 

Markets,  Hierarchies,  and  Networks 

The  specialized  support  required  for  a  project  often  conjures  up  images  of 
hierarchical  organizations  that  integrate  these  specialties  together  for  a  common  purpose. 
Alternatively,  a  project  might  be  accomplished  through  the  marketplace  where  products  and 
services  are  efficiently  offered  to  those  who  have  the  highest  willingness  to  pay.  Ronald 
Coase’s  early  work  on  transaction  costs  compared  firms  and  markets  as  alternatives  to  one 
another.  According  to  Coase,  firms  resorted  to  hierarchy  when  it  was  less  costly  compared 
establishing  and  monitoring  individual  contracts  in  a  market.  The  growth  of  the  firm  was 
balanced  with  the  increasing  expenses  to  organize  a  larger  labor  force  due  to  diminishing 
marginal  returns.  Eventually,  the  cost  of  an  additional  transaction  within  the  firm  was  equal 
to  the  cost  of  contracting  in  the  marketplace  for  the  same  goods  or  services  (Coase,  1937). 

Powell  introduced  the  concept  that  a  network  existed  between  a  self-forming 
marketplace  and  a  hierarchical  organization.  He  rejected  the  view  that  networks  are  neither 
part  of  a  market-to-hierarchy  continuum,  nor  do  they  represent  a  hybrid  form  of  hierarchy. 

As  evidence,  Powell  offered  two  examples  that  pointed  to  the  existence  of  networks.  He 
noted  the  blurring  of  the  boundaries  between  markets  and  inter-organizational 
collaborations,  such  as  cooperative  joint  ventures.  His  second  example  was  the  creation  of 
enduring  relationships  between  hierarchies  and  their  consulting,  law,  and  banking  firms — 
indicating  that  a  network  form  of  governance  existed  between  these  organizations  (1990). 
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Defining  Networks 

Why  Network? 

Before  delving  into  the  definitions  of  a  network,  it  is  worth  noting  the  inherent 
strengths  and  weaknesses  of  each  form  of  organization.  Markets  are  ideal  for  simple 
transactions  in  which  inputs  and  outputs  are  measurable  and  are  not  based  on  a  number  of 
contingencies.  Cease  (1937,  p.  287)  described  the  marketplace  as:  "under  no  central 
control  [...]  supply  is  adjusted  to  demand,  and  production  to  consumption." 

Hierarchies  evolved  to  control  the  specialized  inputs  needed  to  produce  complex 
products  or  services  for  which  the  inputs  may  not  be  available  in  the  commercial 
marketplace.  Coase  (1937)  used  the  classic  example  of  specialized  labor  where  a  firm 
chose  to  employ  an  individual  with  specific  skills,  thereby  internalizing  the  uncertainties 
associated  with  inputs.  Additionally,  the  firm  would  also  observe  that  person’s  work  on  the 
job  and  make  adjustments  (Williamson,  1973).  Therefore,  hierarchies  excel  when  inputs 
have  uncertainty,  since  they  allow  internal  observation  and  adjustment  during  the  course  of 


business. 


Networks  are  adept  at  handling  uncertainty  associated  with  both  inputs  and  outputs. 
O'Toole  (1997)  described  uncertainty  as  leading  to  wicked  problems  that  cannot  be  divided 
into  tasks  that  are  isolated  from  each  other.  Powell  agreed  that  networks  form  as 
organizations  choose  to  pool  resources  to  manage  uncertainties,  thereby  creating 
interdependencies  from  which  a  firm  cannot  easily  walk  away.  He  elaborated  that  networks 
are  particularly  adept  at  exchanging  resources  that  are  difficult  to  measure,  such  as  "know¬ 
how,  technological  capability,  a  particular  approach  or  style  of  production"  (1990,  p.  304). 

Network  Definition 

While  a  network  is  fairly  well  understood  in  today’s  society,  such  familiarity  with 
networks  may  lead  to  a  variety  of  definitions.  The  most  straight  forward  definition  of  a 
network  comes  from  sociology.  Borgatti  and  Foster  (2003,  p.  992)  described  this  type  of 
governance  this  way:  “A  network  is  a  set  of  actors  connected  by  a  set  of  ties.”  Marsden  and 
Lin  (1982)  and  Knoke  and  Kuklinski  (1991)  emphasized  persistent  relationships  among 
actors,  focusing  on  their  relationships  rather  than  the  actors  themselves  or  the  groups  to 
which  they  belong.  Whereas  an  actor  continues  to  exist  apart  from  the  network,  a  network 
does  not  exist  without  the  relationship  between  the  actors. 

Another  example  of  networks  comes  from  the  field  of  public  administration  where 
networks  are  used  among  government,  non-government,  and  private  agencies.  Kickert, 
Klijn,  and  Koppenjan  (1997,  p.  6)  described  networks  as  “stable  patterns  of  social  relations 
between  interdependent  actors,  which  take  shape  around  policy  problems  and/or  policy 
programmes.”  This  definition  is  broad,  spanning  coalitions  of  intergovernmental  and  non¬ 
governmental  actors  organized  around  both  issues  and  delivery  of  public  goods  and 
services. 

This  report  will  utilize  the  Kickert,  et  al.  (1997)  definition  of  networks  in  which  actors 
are  dependent  upon  one  another;  there  are  lasting,  stable  relationships;  and  the  network  is 
formed  around  a  policy  or  project.  In  comparing  this  definition  with  others,  Klijn  (1997) 
identified  three  characteristics  of  networks: 

■  Networks  form  due  to  interdependencies  between  actors. 

■  Networks  consist  of  multiple  actors  who  have  their  own  objectives. 
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■  Networks  consist  of  the  lasting  relationships  between  the  various  actors. 

The  first  condition  for  a  network  is  interdependencies.  Klijn  (1997)  suggested 
resource  dependency  is  a  key  driver  of  lasting  relationships  since  organizations  require 
exchange  of  capital,  personnel,  and  knowledge  with  other  organizations.  Powell  (1990)  and 
Jones,  Hesterly,  and  Borgatti  (1997)  similarly  emphasized  actors  within  networks  performing 
complex  exchanges  and  transactions  using  trust  and  norms  rather  than  market-driven, 
legally  enforceable  contracts. 

Again,  the  condition  for  more  than  one  actor  comes  into  the  definition  with  the  added 
criteria  that  each  has  his/her  own  objectives.  Scharpf  (1978)  concluded  that  within 
government,  there  is  no  single  actor  and  no  unifying  goal.  Instead,  policy  was  a  result  of 
interactions  among  multiple  actors  in  which  coordination  is  achieved  through  exchanges  of 
material,  information,  and  legitimacy. 

The  final  characteristic  of  networks  is  that  they  are  composed  of  lasting  relationships 
among  the  actors.  Klijn  (1997)  and  Jones,  Hesterly  and  Borgatti  (1997)  concluded  that 
relationship  patterns  in  a  network  are  defined  according  to  their  frequency  over  time. 
Repeated  interactions  strengthen  the  relationship.  As  a  pattern  of  behavior  develops  during 
on-going  interactions,  actors  will  begin  to  understand  who  they  can  trust  and  who  they 
cannot  trust.  Therefore,  the  basis  for  the  network  is  the  willingness  to  establish 
interdependency  based  on  that  frequent,  lasting  relationship. 

Network  Analysis 

Network  Structure 

In  analyzing  a  network,  the  individuals  within  the  network  are  not  as  important  as  the 
relationships  between  them.  Since  networks  imply  interactions  in  which  no  one  individual 
has  all  the  resources  to  solve  a  problem,  the  dyadic  relationship  is  the  basic  unit  of 
structure.  At  the  next  higher  level  of  analysis,  the  network  as  a  whole  will  determine  the 
success  of  outcomes.  How  the  dyadic  relationships  are  arranged  to  form  a  network  count  in 
achieving  a  result.  Therefore,  structure  determines  how  the  group  as  a  whole  will  perform. 


Figure  2.  Asymmetric  Informational  Network 

(Knoke,  1990,  p.  237) 
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Table  1.  Matrix  Representation  of  an  Asymmetric  Network 

(Knoke,  1990,  p.  237) 


A 

B 

C 

D 

E 

F 

G 

H 

1 

J 

Total 

A 

0 

0 

0 

1 

0 

0 

0 

0 

0 

1 

B 

1 

1 

1 

0 

0 

0 

0 

0 

0 

3 

C 

1 

1 

1 

0 

0 

0 

0 

0 

0 

3 

D 

1 

1 

1 

0 

0 

0 

0 

0 

0 

3 

E 

1 

0 

0 

0 

1 

1 

1 

0 

0 

4 

F 

0 

0 

0 

0 

1 

0 

0 

0 

0 

1 

G 

0 

0 

0 

0 

1 

0 

0 

0 

0 

1 

H 

0 

0 

0 

0 

1 

0 

0 

0 

0 

1 

1 

0 

0 

0 

0 

0 

0 

1 

0 

0 

1 

J 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Total 

4 

2 

2 

2 

4 

1 

2 

1 

0 

0 

18 

To  illustrate  the  concepts  of  measuring  information  flow  in  a  network,  a  hypothetical 
example  of  a  network  in  which  actors  exchange  information  asymmetrically  is  shown  in 
Figure  2.  The  arrows  depict  the  flow  of  information.  An  adjacency  matrix  is  used  to 
represent  this  information  flow  from  actors  to  one  another.  The  number  one  in  a  row 
represents  transmitting  information  from  the  actor  in  the  row  to  the  actor  in  the  column, 
whereas  a  zero  indicates  that  no  information  is  transmitted.  The  number  one  in  a  column 
represents  receipt  of  information,  and  a  zero  represents  no  information  receipt.  Knoke 
(1990)  developed  the  above  matrix  in  Table  1 ,  concluding  from  the  totals  for  the  columns 
and  rows  that  A  receives  the  information  from  more  sources,  and  E  transmits  information  to 
the  most  actors. 

Actors’  Positions  within  the  Network 

Switching  from  the  network  back  to  the  individual  actor  as  a  unit  of  analysis,  the 
above  tools  also  allow  an  assessment  of  how  the  actor  fits  into  the  structure  of  the  network. 
Freeman  (1977)  introduced  measures  of  a  node's  centrality  based  on  his  definition  of 
position  centrality:  "the  degree  they  stand  between  others,  and  can  therefore  facilitate, 
impede  or  bias  the  transmission  of  messages."  These  nodes  control  the  information  flow  in 
the  network  more  than  others. 

Centrality  appears  to  be  directly  correlated  with  the  efficiency  of  the  network  and  the 
power  of  the  individual  who  is  more  central.  Freeman  (1977)  applied  centrality  measures  to 
other  studies  of  communication  in  small  group  settings,  and  concluded  that  centrality  was 
related  to  solving  problems  with  speed,  efficiency  and  personal  satisfaction.  Likewise, 
Krackhardt's  (1990)  work  correlated  Freeman's  measures  of  centrality  to  perceived  power  in 
a  small,  entrepreneurial  organization. 

Relating  Network  Structure  to  Network  Effectiveness 

From  the  perspective  of  the  network  as  a  whole,  a  definition  of  network  effectiveness 
must  be  defined  on  multiple  levels  across  multiple  agencies.  Provan  and  Milward  (2001) 
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offer  the  community,  or  area  the  network  serves,  the  network  itself,  and  the  organization  and 
participants  as  the  levels  among  which  a  network  should  be  evaluated  to  satisfy  multiple 
stakeholder  perspectives.  Their  empirical  study  developed  the  following  conclusions: 

■  Networks  are  more  effective  when  they  are  integrated  through  centralization, 
although  networks  that  are  integrated  through  a  core  agency  and  integrated  through 
dense  links  among  members  will  be  less  effective  than  those  are  integrated  through 
a  core  agency  alone. 

■  Networks  are  most  effective  when  external  controls  are  directly  applied,  rather  than 
applied  through  an  agency. 

■  Networks  are  most  effective  when  a  degree  of  stability  is  achieved,  especially  when 
the  stability  and  uncertainty  impacts  clients. 

■  When  the  above  conditions  are  optimal,  resource  scarcity  will  limit  the  effectiveness 
of  the  network.  Conversely,  resource  abundance  allows  the  network  to  range  from 
low  to  high  effectiveness,  depending  on  the  conditions  above. 

Analysis 

Application  of  the  network  Perspective  to  weapon  system 
acquisition 

Chapter  II  described  the  acquisition  system  and  its  formal  hierarchical  operating 
structure.  Chapter  III  introduced  the  network  perspective  and  its  basic  assumptions  and 
methodology.  This  chapter  draws  on  the  data  from  the  APAT  process  model  and  concludes 
that  the  acquisition  system  has  network-like  properties.  The  implications  of  the  acquisition 
system’s  network  characteristics  are  explored  in  terms  of  acquisition  governance. 

Interdependencies  between  Actors 

One  of  the  key  characteristics  of  a  network  is  the  relationships  between  the  actors. 
Interdependencies  between  actors  are  the  basis  for  the  formation  of  networks  (Klijn,  1997; 
Powell,  1990;  Jones,  Hesterly  &  Borgatti,  1997).  The  interdependencies  are  based  on  the 
exchange  of  resources,  and  develop  in  situations  in  which  the  actors  need  capital, 
personnel,  and  knowledge  to  accomplish  their  objectives  (Klijn,  1997). 

To  deliver  a  weapon  system,  numerous  actors  are  involved,  as  shown  in  the 
relationship  matrices  in  Appendix  B.  One  of  the  key  interdependencies  during  the 
acquisition  process  is  the  exchange  of  knowledge  between  actors.  During  the  first  three 
phases  of  the  acquisition  process,  knowledge  about  what  you  need  to  buy  and  how  the 
system  should  be  designed  to  meet  that  need  is  the  focus  of  the  activities.  As  shown  in 
Appendix  A,  Process  1 .0,  the  outputs  of  the  Concept  Refinement  phase  include  an 
approved  Course  of  Action,  identification  of  resources  needed  for  the  next  phase,  approved 
milestone  decision  documents,  a  signed  acquisition-decision  memorandum,  and  a 
technology-development  strategy.  All  of  this  knowledge  is  based  on  the  collaborations 
among  the  stakeholders  during  each  activity. 

The  interdependencies  between  actors  for  Concept  Refinement  are  modeled  in 
Figure  3.  For  modeling  purposes,  the  interactions  are  assumed  to  be  two-way,  directed 
collaborations.  The  relationships  are  those  that  are  specified  in  the  Concept  Refinement 
processes  or  may  be  inferred  from  the  type  of  documents  that  are  approved  during  that 
phase.  For  example,  approval  of  a  Test  Evaluation  Master  Plan  for  a  large  program 
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requires  an  OSD  (DOT&E)  signature.  Of  course,  these  are  not  the  only  relationships  that  a 
program  might  need  to  carry  out  the  objectives  of  this  phase  of  the  acquisition  cycle.  This  is 
a  minimum  set  that  one  would  expect  to  see  for  any  major  acquisition  program. 

The  diagram  shown  in  Figure  3  illustrates  the  interdependencies  required  to  define 
the  weapon  system  concept,  select  the  course  of  action,  and  prepare  for  the  Technology 
Development  phase.  As  one  could  guess  based  on  the  description  of  responsibilities  in 
JCIDS  and  the  DoD  5000  series  regulations,  the  lead  acquisition  organization  program 
manager  (node  4),  the  MAJCOM  requirements  organization  (node  2),  and  the  milestone 
decision  authority  (node  5)  have  critical  roles  during  this  phase.  Freeman's  measure  of 
degree  centrality  (1977)  for  those  nodes  is  relatively  higher  indicating  the  probability  that 
they  will  control  resources  in  the  network. 

Graphically,  Figure  3  portrays  the  collaboration  required  with  the  other  22  actors  to 
accomplish  the  outputs  of  this  acquisition  phase.  Individually,  the  lead  acquisition 
organization,  the  MAJCOM  requirements  organization,  and  the  milestone  decision  authority 
do  not  interface  with  all  of  the  other  actors.  The  spreadsheet  in  Appendix  B  for  the  Concept 
Refinement  interactions  denotes  the  lead  acquisition  organization  collaborating  with  18  other 
actors.  Of  the  seven  actors  with  which  the  lead  acquisition  organization  does  not  interface, 
the  program  manager  must  rely  on  other  organizations  to  gather  information  from  those 
parts  of  the  network. 


Figure  3.  Concept  Refinement  Interdependencies 


IS 


Given  a  weapon  system  concept,  the  purpose  of  the  Technology  Development 
phase  of  the  acquisition  process  is  to  mature  key  technologies  and  to  plan  for  weapon 
system  development.  These  two  activities  require  a  diverse  set  of  interdependencies. 
Maturing  technology  requires  an  in-depth  understanding  of  the  concept  and  system 
architecture  as  well  as  a  diverse  network  of  technology  providers.  Furthermore,  the  program 
must  define  the  capability  needs  in  the  Capabilities  Development  Document  (CDD).  Along 
with  the  capability  needs,  operational,  support,  maintenance,  and  interoperability  concepts 
must  be  refined  so  the  weapon  system  may  be  designed  with  these  plans  in  mind.  The 
acquisition  systems  engineering,  test,  logistics,  contracting,  and  financial-management 
communities  collaborate  with  the  warfighters  to  translate  concepts  into  an  executable 
acquisition  program. 
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To  understand  these  interactions,  this  analysis  focuses  on  process  2.1.2,  Identify 
Technologies  for  Maturation,  process  2.1.3,  Define  Technology  Maturation  Plan,  and 
process  2.5,  Develop  and  Prepare  Documents  for  Milestone  B,  which  the  APAT  model 
decomposed  as  noted  in  Appendix  A.  The  diagram  of  the  interdependencies  for  this  phase 
is  illustrated  in  Figure  4,  while  the  matrix-view  is  in  Appendix  B. 

The  diagram  in  Figure  4  reveals  that  there  are  28  actors  involved  in  the  acquisition 
program,  an  increase  from  the  Concept-refinement  phase.  Based  on  degree  centrality,  the 
lead  acquisition  organization/program  manager  (node  4)  remains  the  most  central  actor, 
maintaining  many  of  the  relationships  from  the  previous  phase.  Likewise,  the  MAJCOM 
requirements  organization  (node  2)  and  the  milestone  decision  authority  (node  5)  also 
maintain  their  central  role.  A  number  of  other  actors  at  the  OSD  and  service-level  become 
more  prominent,  as  demonstrated  by  their  degree  centrality.  The  network  relies  on 
relationships  with  these  actors  to  provide  guidance  and  priorities  that  shape  the  program 
from  an  operational,  acquisition  strategy,  and  budget  perspective.  Therefore,  the  influence 
of  the  key  actors  is  still  great,  but  there  are  many  relationships  developing  during  this  phase 
that  are  beyond  the  control  of  the  key  actors. 

Figure  4.  Technology  Development  Planning/Milestone  Interdependencies 


During  System  Development  and  Demonstration,  the  critical  activities  include 
allocating  requirements  and  developing  a  design,  testing  the  design,  and  preparing  for 
production  and  fielding  of  the  system.  This  analysis  focuses  on  process  3.1 ,  Manage  the 
Program  and  process  3.2.3,  Develop  Detailed  Design  from  the  APAT  effort  in  Appendix  A. 
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Figure  5.  System  Development  and  Demonstration  Interdependencies 


12 


The  diagram  in  Figure  5  depicts  a  less  dense,  decentralized  network.  In  terms  of 
degree  centrality,  the  program  manager  (node  4)  is  still  the  most  central  actor,  although  the 
MAJCOM  requirements  organization  (node  2)  is  now  less  central  than  the  contractor  (node 
10)  in  influencing  the  network.  The  rise  of  the  contractor's  centrality  indicates  the 
importance  of  the  contractor’s  information  to  the  network  in  a  monopolistic  environment. 

This  measure  of  centrality  for  the  sole  non-governmental  actor  is  of  interest  to  those  who 
want  to  influence  the  outcome  of  the  network  at  the  community,  network,  and  organizational 
levels  of  analysis. 

Multiple,  Independent  Actors  Formed  around  a  Project 

Another  characteristic  of  a  network  is  that  there  is  more  than  one  actor  who  shares 
some  common  attribute  that  forms  the  context  of  their  relationship.  Using  the  types  of 
network  relationships  from  Knoke  and  Kuklinski  (1991),  the  actors  involved  in  concept 
refinement  would  share  several  types  of  relationships.  Since  information  is  a  key  resource, 
many  relationships  are  communication  relations.  Relationships  with  industry  might  be 
described  in  transactional  terms,  where  dollars  are  expended  so  resources  can  be  utilized  to 
help  gather  information  on  different  acquisition  concepts.  Finally,  authority/power  relations 
exist  among  the  relationships.  Process  1 .1  in  Appendix  A  describes  the  controls  on  the 
process  from  the  Congressional,  OSD,  and  service  level.  These  controls  may  be  targeted 
specifically  at  a  program,  such  as  when  Congress  earmarks  an  appropriation  for  a  specific 
program. 

One  of  the  key  questions  is  whether  the  actors  remain  independent  to  pursue  their 
own  objectives  for  the  project.  As  noted  above,  there  are  authority/power  relations  exerted 
on  the  program.  In  fact,  the  lead  acquisition  organization  program  manager  works  for  the 
service  acquisition  executive,  typically  through  the  PEO  as  an  intermediate  supervisor. 

Many  of  the  actors,  however,  do  not  work  for  one  another.  Congress  clearly  does  not  work 
for  the  program  manager,  and  the  converse  is  also  true.  In  addition,  key  collaborators  such 
as  the  MAJCOM  and  the  lead  acquisition  organization  do  not  work  for  one  another  even 
though  they  are  in  the  same  service.  If  the  lead  acquisition  organization  and  the  MAJCOM 
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requirements  organization  had  a  dispute,  they  would  have  to  resolve  it  at  the  Chief  of  Staff 
of  the  Air  Force/Secretary  of  the  Air  Force  level.  Issues  are  not  resolved  typically  at  that 
level.  Instead,  the  actors  utilize  their  relationship  with  one  another  to  collaborate  and  work 
through  issues. 

Lasting,  Stable  Relationships  between  Actors 

The  final  characteristic  to  be  analyzed  is  the  pattern  of  relationships  between  actors 
over  time.  Again,  the  literature  stresses  the  importance  of  this  characteristic  based  on  the 
need  to  strengthen  relationships  (Klijn,  1997).  In  long-term  acquisition  programs  with  both 
complexity  and  uncertainty,  this  characteristic  is  important  to  allow  actors  to  establish  trust 
with  one  another.  This  trust  enables  actors  to  make  transaction-specific  investments  that 
will  further  the  objectives  of  both  the  actors  and  the  network. 

To  examine  this  variable,  the  set  of  actors  in  the  first  three  acquisition  phases  were 
analyzed  to  determine  if  the  relationship  spanned  multiple  acquisition  phases — which  could 
last  from  a  couple  of  years  to  over  a  decade.  The  analysis  is  inexact  since  only  select 
processes  from  the  Technology  Development  phase  and  System  Development  and 
Demonstration  phase  were  analyzed.  Nonetheless,  a  group  of  10  actors  form  13  enduring 
relationships  that  span  the  formation  and  development  of  an  acquisition  program.  This 
group  of  key  players  and  their  relationships  are  displayed  in  Appendix  B. 

Fligh-degree  centrality  among  this  core  group  denotes  the  actors  who  continually 
control  resources  over  time.  Not  surprisingly,  the  program  office  has  the  highest  degree 
centrality  within  this  persistent  group  of  core  actors.  Interestingly,  the  MAJCOM  budget 
organization  and  modernization  budget  integrator  on  Air  Staff,  SAF/AQX,  also  have  high- 
degree  centrality — stemming  from  their  control  over  the  fiscal  resources  needed  to  execute 
the  acquisition  program. 

Network  Governance 

A  network  view  of  acquisition  allows  an  analyst  to  examine  outcomes  and 
management  strategies  in  a  new  way.  Rather  than  focusing  on  accountability,  the  focus 
shifts  to  understanding  how  to  enable  the  network  as  a  whole  to  create  greater  value.  As 
Provan  and  Milward  (2001)  suggested,  the  effectiveness  of  the  network  ought  to  be 
analyzed  at  the  community,  network,  and  participant  level.  Understanding  the  outcomes 
desired  from  acquisition  programs  across  the  Congressional  and  warfighting  community,  the 
acquisition  community,  and  the  individual  organizations  within  the  network  allows  a  holistic 
analysis  of  how  the  network  ought  to  be  structured  to  accomplish  these  desires. 

A  review  of  the  data  in  Appendix  B  supports  the  conclusion  that  the  Lead  Acquisition 
Organization/Program  Manager  is  the  most  central  actor  within  the  acquisition  process  in 
terms  of  degree  centrality.  Furthermore,  this  actor  has  the  greatest  range  of  relationships, 
brokering  information  from  the  warfighter,  budget  community,  technology  community,  and 
contractor.  This  places  the  Program  Manager  in  a  very  important  position  in  the  network. 

Not  all  program  managers  perform  equally.  Some  may  be  unable  to  stabilize  their 
inherently  unstable  networks.  Other  managers  may  have  perfectly  adequate  networks,  but 
the  manager  is  unable  to  understand  how  to  manage  in  a  network.  Whatever  the  case, 
understanding  the  structure  of  the  network  should  enable  program  managers  to  understand 
the  environment  within  which  successful  programs  are  executed. 

Further,  an  understanding  of  the  network  allows  an  analysis  of  second-order  effects 
due  to  changes  in  the  network.  Provan  and  Milward  (1995)  concluded  that  resource  scarcity 
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would  limit  the  effectiveness  of  any  network.  When  resources  are  adequate,  however, 
factors  such  as  centralization,  direct  external  controls,  and  stability  may  also  affect  the 
outcomes  of  networks.  An  understanding  of  the  structure  of  the  acquisition  program 
network  would  allow  an  analysis  of  the  effects  of  changes  using  modeling.  The  resultant 
outcomes  could  be  analyzed  at  the  participant,  network,  and  community  level.  In  other 
words,  a  network  view  of  acquisition  would  allow  individual  participants  to  understand  how 
their  outcomes  and  the  network's  outcomes  would  be  affected  by  the  continuing  change  in 
policy,  resources,  and  players  in  the  acquisition  system. 

Conclusions  and  Recommendations 

Conclusions 

Research  Question 

The  focus  of  this  paper  was  to  answer  the  following  research  question:  does  the 
DoD  weapon  system  acquisition  process  behave  as  a  network? 

The  characterization  of  the  "Big  A"  acquisition  system  as  a  complex  interaction  of  the 
JCIDS  subsystem,  the  PPBE  subsystem,  and  the  defense  acquisition  subsystem  identified 
multiple  stakeholders  who  value  different  outcomes.  Each  of  these  players  attempts  to 
utilize  some  form  of  hierarchy  to  break  down  tasks  and  assign  responsibility  to  ensure  task 
accomplishment. 

However,  the  APAT  process  model  revealed  a  more  complex,  interactive  process 
among  the  JCIDS,  PPBE,  and  the  acquisition  subsystems.  Appendix  B  portrays  the  key 
players  in  the  first  three  phases  of  the  acquisition  cycle.  An  analysis  of  these  players 
reveals  that  many  do  not  work  for  one  another  and  may  have  differing  objectives. 
Furthermore,  examination  of  the  key  activities  within  the  Concept  Refinement,  Technology 
Development,  and  System  Development  and  Demonstration  phases,  and  the  interactions  of 
the  key  players  who  were  involved  in  the  controls,  inputs,  activities,  and  outputs  of  each 
subsystem,  revealed  key  interdependencies  and  long,  stable  relationships  among 
independent  actors.  This  analysis  led  to  the  conclusion  that  weapon  system  acquisition  can 
be  conceptualized  as  a  network. 

Further  Refinements 

Analysis  of  the  APAT  process  model  data  also  led  to  an  understanding  that  centrality 
is  not  equally  distributed  within  the  network.  The  lead  acquisition  organizations/program 
manager  is  a  central  figure  who  has  the  greatest  number  of  relationships  and  is  most  central 
to  the  network  measured  in  terms  of  degree  of  centrality.  Despite  the  program  manager's 
lack  of  a  high  position  within  a  hierarchical  model,  network  analysis  reveals  that  the  program 
manager  has  the  greatest  number  of  contacts  and  interactions  within  the  network. 

Additionally,  there  Is  a  core  group  of  actors  who  have  a  persistent  set  of  relationships 
during  the  early,  critical  stages  of  the  acquisition  process.  While  the  program  manager  is 
well-placed  within  this  core  group,  there  are  other  important  actors  who  deal  with  budgets 
and  have  sustained  relationships  over  time.  Understanding  the  structure  of  this  group  and 
their  relationships  with  the  rest  of  the  network  will  be  important  in  helping  the  acquisition 
community  develop  strategies  to  govern  the  network  and  influence  changes  for  improved 
network  performance  and  outcomes. 
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Recommendations 


Validate  the  Model 

First,  the  data  gathered  in  the  APAT  model  were  intended  to  serve  as  a  framework  to 
understand  the  current  acquisition  process  as  it  applies  to  a  majority  of  programs.  The 
scope  of  the  data-gathering  process  limited  the  ability  to  focus  on  all  interactions. 

Therefore,  activities  such  as  milestone  decisions  were  described  as  an  exercise  in 
document  writing.  Those  involved  in  the  APAT  effort  recognized  that  the  documents 
generated  for  a  milestone  decision  were  actually  the  culmination  of  a  set  of  interactions  to 
gather  data  and  develop  a  strategy  for  a  particular  portion  of  the  acquisition  program.  For 
this  effort,  the  official  who  approved  the  document  and  the  program  office  WIPT  were 
assumed  to  be  the  only  participants.  This  is,  in  fact,  probably  not  true.  Participants  might 
include  other  organizations,  depending  on  the  subject  matter  of  the  program  and  local 
procedures. 

Therefore,  the  model  serves  as  a  good  first  step  to  begin  to  explore  certain 
interactions  within  the  acquisition  system.  If  a  certain  set  of  interactions  or  a  set  of  actors 
are  of  interest,  gathering  more  detailed  data  would  be  valuable  to  further  the  understanding 
of  the  network  and  to  validate  the  model. 

Network  Framework  to  Study  Improved  Outcomes 

The  data-gathering  effort  for  the  APAT  model  was  not  prescriptive.  While  the 
sponsors  of  the  effort  were  interested  in  recognizing  areas  for  improvement,  the  model  was 
meant  to  describe  the  current  process.  There  are  reasons  for  the  patterns  of  relationships 
established  in  the  model,  but  there  also  may  be  improved  ways  of  interacting. 

Indeed,  the  network  model,  once  validated,  could  be  utilized  as  a  framework  to 
assess  program  success.  Those  who  control  acquisition  policy  or  who  participate  in 
acquisition  programs  likely  would  be  interested  in  studying  how  the  networks  of  these 
programs  of  interest  differ  from  the  norm.  DoD  Directive  5000.1  gives  the  program  manager 
and  milestone  decision  authority  flexibility  to  decide  what  the  correct  set  of  activities  and 
relationships  should  be  for  a  particular  acquisition  program.  Studying  network  models  of 
similar  programs  might  enable  decision-makers  to  tailor  their  efforts  and  focus  resources  on 
valuable  relationships.  Alternatively,  acquisition  strategies  could  be  modeled  to  discover  if 
information  flows  efficiently  and  effectively  given  several  scenarios  for  organizing  a  program. 

Simulate  Changes  to  the  Acquisition  System 

Of  course,  there  are  number  of  challenges  within  the  acquisition  process. 
Consistently  delivering  cost,  schedule,  and  performance  is  rare  as  Augustine  and  Fabini 
(1983),  Jones  (1996),  and  McNutt  (1998)  agreed.  Improving  consistency  of  the  system  has 
spawned  a  number  of  changes — some  of  which  are  initially  declared  successful,  only  to  be 
later  discredited  for  their  "unintended  consequences."  An  example  is  the  initiative  to  give 
the  contractor  Total  System  Performance  Responsibility.  This  initiative  gave  the  contractor 
more  flexibility  and  responsibility  for  the  performance  of  the  acquisition  program. 
Unfortunately,  the  effects  of  this  change  were  probably  not  studied  using  a  network  analysis. 
The  decision-makers  acted  upon  the  ideology  that  the  marketplace  solved  all  their  problems. 

A  number  of  changes  to  the  acquisition  system  are  being  considered  today.  JCIDS 
mandates  that  programs  have  been  have  a  Net-ready  Key  Performance  Parameter 
(Chairman  of  the  Joint  Chiefs  of  Staff,  2005).  This  attempt  to  build  a  communication  system 
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by  mandating  interoperability  from  those  who  will  utilize  the  system  is  much  like  the  chicken 
and  the  egg  conundrum.  First,  the  architecture  of  the  network  must  have  some  definition. 
Those  who  are  developing  a  network  and  the  users  of  the  network  must  collaborate  to  solve 
this  problem.  Clearly,  a  network  analysis  to  identify  who  is  involved  and  how  they  are 
collaborating  would  be  more  beneficial  than  mandating  a  change  and  hoping  that  those 
actors  in  the  network  would  comply. 

Summary 

Networks  describe  both  formal  and  informal  ways  of  getting  things  done  in  the 
acquisition  system.  The  marketplace  rarely  delivers  what  the  DoD  needs  at  the  quantity  that 
it  is  needed.  Some  commodities  may  be  purchased  in  the  marketplace,  but  the 
uncertainties  associated  with  DoD  needs  do  not  allow  firms  to  match  their  supply  to 
demand.  Also,  many  of  the  DoD's  needs  are  based  on  interoperability  between  programs 
that  must  be  defined  before  the  market  can  react  to  this  need.  The  largest  transactions, 
which  involve  the  lion's  share  of  the  modernization  budget,  rely  on  the  interactions  between 
JCIDS,  PPBE,  and  the  acquisition  system.  A  hierarchy  exists  to  account  for  the  resources 
input  into  the  process.  However,  the  complexities  and  dynamic  nature  of  the  process  can 
best  be  described  as  a  network  of  actors  who  use  their  relationships  to  affect  outcomes. 

Would  Glenn  Curtiss  recognize  this  network  that  delivers  today's  innovative  stealth 
aircraft,  advanced  combat  systems,  and  ships?  He  probably  would.  If  you  brought  Mr. 
Curtiss  into  a  meeting  with  a  program  manager,  MAJCOM  requirements  officer,  and  a 
contractor,  he  would  feel  right  at  home.  Mr.  Curtiss  was  no  stranger  to  hierarchies  given  the 
size  of  the  Curtiss  Aircraft  Company.  Nonetheless,  he  knew  that  innovation  occurs  when  a 
network  of  collaborators  shares  ideas  to  solve  their  common  problems. 
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1.1  Determine  Resource  Needs  for  Concept  Refinement 
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1.2.1  Modify  Potential  Alternatives 
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1 .2.2  Develop  Analytical  Approach 


0) 

Ui 

Ui 


06 


CO 


T3  i2  CD 

■g  §  o 

Z  CD  CD 

^■E  O 

E  ^  2 

—  CD  Q- 

O 


JD  LLI 

^  O 

O  ^  CO 
0^0. 
o  o 

CO  ^ 

CO  ^ 

.E  Oi 
P  > 

o  CD 
8  0^ 

O 


(0 

o 


o 

< 


CL 

o 


O 

^  ^  o 

^  .  CD 


:-E  o 


3 

a 

3 

o 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  -  232  - 


Modify/  Refine  Models 


1.2.3  Perform  Analyses  &  Evaluate  Results 


c 

0 

E 

0 

0 

0 

0 

0 

< 

0 

q: 


0 

■E  0 
0  CL 

O  p) 
0  ^ 

0  o3 

E  0 

0  LJJ 
0  O 

si 


Od  +3 


8  ^ 
C  = 
0  ^ 


0 

> 


-1— » 

0 

C 

0 

0 

■0 

CL 

o 

c 

O) 

0 

O 

0 

E 

"c 

0 

oa 

£ 

E 

0 

t 

0 

LJJ 

0 

CL 

0 

0 

O 

0 

0 

< 

C 

_0 

CL 

■D 

CO 


o6 


0 


CO 


> 

■■G 

0 

In’ 

O  ^ 
o6  0 


2  0 


0 

Q.  0 

OE 

oa  ^ 


0 

0 

> 

0 

E  CNj 
0  ^ 

^  § 

0  M- 

II 

-a 

c 

0 

O 


c 

0 

Q 

<-o 

CO 


oa 


0 

'  "5 
o 


CO 

■D 

0 

> 

P 


0 
0 
Q.  0 

9-  Q 


CL  -.6 
O  ^ 

oa  o  - 
o 

^  Q  ■ 


0  E 
■—  P 
P  0 

^OE 

5  oS  ^ 


Q.  0 

§1 

o  O 


3 

Q. 

£ 


0 

0 

'0 

o 

O 

0 

E 

c 

< 

£ 

0 

t 

0 

0 

0 

CL 

C 

0 

CO 

> 

CO 

tj 

c\i 

tt 

UJ 

(0 

(0 

o 

o 

o 


0 

0 

O 

■D 

E 

o 

O  0 

t 

CO  ^ 

0 

T5  2 

CL 

C  E 

CM 

0  < 

CO 

0 

CM 

q: 

E 

_0 

0 

C 

< 

0 

0 

CL 

0 

0 

C 

CO 

0 

CM 

_> 

tj 

0 

y= 

UJ 

0  0 
8g 
°  i 

J5  < 

^  0 

0  0 

rr  ^ 

ro  £ 

5  m 


0 

'0 

0 

oa  '0 

0 

C 

< 

0  0 

o  < 

0 

fV 

E  P 

LL. 

E 

P-g 

0  ^ 
CL  O 

0 

CO 

CL 

(0 

0) 


-S 


^  0 

^  S 
■  >  0 
"m  ^ 
-E  0  C 

^  i"! 
8< 


a 

3 

o 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE 


-233- 


1.3  Define  Alternative  Courses  of  Action  COA(s) 
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1 .4  Repare  for  Technology  De^opnnent 
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Metrics  Metrics  -  None  Ictentified 


2.1.3  Define  Technology  Maturation  Plan 
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2.5  Develop  and  Prepare  Milestone  B  Documents 
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Appendix  B.  Acquisition  Networks 


A.  Concept  Refinement  Network 
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B.  Technology  Development  Planning/Milestone  Network 
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(26)  DAB  Participants 

(27) DARPA 

(28) AFAcqExec 


C.  System  Development  and  Demonstration 
Management/Design  Network 
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D.  Persistent  Relationships  from  Concept  Refinement  Through 
System  Development  and  Demonstration 
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Developing  Collaborative  Capacity:  A  Diagnostic  Model 
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Abstract 

Federal  Acquisition  Reform  acknowledges  the  importance  of  effective  collaboration 
among  participating  organizations.  However,  both  research  and  practical  experience  have 
shown  that  inter-organizational  collaboration  can  be  difficult  to  achieve.  This  research  builds 
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on  a  model  developed  by  the  authors  with  homeland  security  organizations  that  identified 
enablers  and  barriers  to  collaborative  capacity.  The  focus  of  our  current  research  is  to 
develop  a  diagnostic  mechanism  that  can  be  used  to  improve  that  capability.  The  initial 
conceptual  model  and  research  from  homeland  security  has  been  elaborated  into  an  item 
bank  of  diagnostic  interview  and  survey  questions.  A  diagnostic  process  based  on  the 
established  practices  of  organizational  development  is  offered  to  guide  the  design  and 
application  of  tailored  assessments.  Recommendations  are  given  for  the  use  of  the 
diagnostic  process  to  generate  organization-  or  network-specific  data  that  can  guide  action 
planning  to  improve  collaborative  capacity. 

Introduction 

The  research  presented  at  this  symposium  represents  the  completion  of  our  FY06 
activities  as  well  as  the  plans  and  progress  on  the  FY07  efforts  in  the  development  of  a 
diagnostic  method  to  assess  the  capacity  of  organizations  to  work  collaboratively  with  other 
organizations.  The  first  phase  of  the  research  developed  a  conceptual  model  of  inter- 
organizational  collaboration,  identified  factors  that  enable  or  inhibit  collaboration  and 
established  a  comprehensive  item  bank  of  survey  and  interview  questions  that  could  be 
used  to  diagnose  the  collaborative  capacity  of  an  organization  or  system  of  organizations.  It 
also  outlined  a  process,  based  on  principles  of  organizational  development,  by  which  the 
diagnostic  results  can  be  used  to  design  interventions  to  improve  collaborative  capacity. 

The  second  phase,  which  is  in  progress,  is  field  testing  the  items  as  well  as  the  diagnostic 
process  on  interagency  partnerships  in  both  the  homeland  security  and  acquisition  context. 
The  product  of  Phase  Two  will  be  a  revised  and  validated  diagnostic  instrument  and  field- 
validated  model  for  collaborative  capacity. 

Background 

Federal  Acquisition  Reform  has  consistently  called  for  more  and  better  collaboration 
among  participating  acquisition  agencies  as  well  as  between  the  DoD  and  defense 
contractors.  Specifically,  the  DOD  Directive  5000.1  {The  Defense  Acquisition  System, 
paragraph  E1.2,  Collaboration,  2003)  specifically  states  that  teaming  among  warfighters, 
users,  developers,  acquirers,  technologists,  testers,  budgeters,  and  sustainers  shall  begin 
during  the  capability-needs-definition  phase  of  the  acquisition  lifecycle.  Furthermore,  the 
recent  Defense  Acquisition  Performance  Assessment  (DAPA)  report  recommends  improved 
collaboration  among  acquisition  organizations  as  well  as  between  the  DoD  and  industry. 

The  use  of  Integrated  Product  Teams  (IPTs),  Partnering  relationships,  and  Alpha 
Contracting  processes  are  a  few  examples  of  innovative  arrangements  being  used  in  some 
commands.  As  DAPA  recommendations  are  implemented,  additional  collaboration 
requirements  and  opportunities  will  emerge. 

Collaboration  across  organizations  in  government  and  industry  has  been  found  to 
reduce  litigation,  decrease  costs,  and  increase  innovation  (Mankin,  Cohen  &  Fitzgerald, 
2004).  Flowever,  experience  shows  that  organizations  commonly  fail  when  they  attempt  to 
build  collaborative  relationships.  Among  the  reasons  for  ineffective  collaboration  are: 
diverse  missions,  goals  and  incentives  that  conflict  with  one  another;  histories  of  distrust  that 
are  hard  to  alter;  leaders  who  do  not  actively  support  collaborative  efforts;  and  the  lack  of 
supportive  coordination  systems  and  structures  (US  Government  Accountability  Office, 

2002,  December).  However,  experience  shows  that  inter-organizational  collaboration  can 
be  difficult  at  best.  Our  research  focuses  on  identifying  and  assessing  those  factors  that 
facilitate  or  inhibit  successful  collaboration,  with  the  ultimate  aim  of  guiding  actions  to 
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enhance  the  capacity  of  organizations  to  collaborate  with  each  other  when  appropriate.  We 
define  collaborative  capacity  as  the  ability  of  organizations  to  enter  into,  develop,  and 
sustain  inter-organizational  systems  in  pursuit  of  collective  outcomes. 

Development  of  the  Diagnostic  Model  and  Assessment  Tool 

In  a  series  of  studies,  beginning  with  research  on  homeland  security  organizations 
and  then  reformulated  to  also  address  the  acquisition  context,  we  have  developed  a  general 
framework  for  addressing  the  problem  of  how  interagency  collaborative  capacity  is 
developed  and  maintained  (Hocevar,  Jansen  &  Thomas,  2004;  Hocevar,  Thomas,  & 

Jansen,  2006).  The  model  developed  in  Phase  One  of  our  research  program  identifies 
imperatives  of  successful  collaboration  and  aims  to  assist  organizations  in  diagnosing  their 
collaborative  capacity.  The  focus  of  Phase  Two  was  the  development  of  a  database  of 
interview  and  survey  questions  that  can  be  used  to  tailor  collaborative  capacity  assessments 
to  specific  collaborative  contexts.  The  goal  of  the  diagnostic  is  to  allow  organizations  to 
assess  their  capacity  to  engage  in  collaborative  efforts  and  then  use  the  assessment  results 
to  identify  specific  activities  to  improve  their  collaborative  capacity.  The  survey  and 
interview  questions  were  developed  in  conjunction  with  the  five  dimensions  in  our  model  of 
interagency  collaboration  capacity.  The  dimensions  are  presented  in  more  detail  in  Figure 
1. 


Figure  1.  Factors  Related  to  the  Development  of  Collaborative  Capacity 
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Incentives 

-  Collaboration  as  a 
prerequisite  for  funding  or 
resources 

-  Leadership  support  and 
commitment 

-  Absence  of  competitive 
rivalries 

-  Acknowledged  benefits  of 
collaboration  (e.g.,  shared 
resources) 

-  Competition  for  resources 

-  Territoriality 

-  Organization-level  distrust 

-  Lack  of  mutual  respect 

-  Apathy 

-  Appreciation  of  others’ 

-  Lack  of  competency 

perspectives 
-  Competencies  for 

-  Arrogance,  hostility,  animosity 

People 

collaboration 

-  T  rust 

-  Commitment  and 
motivation 

Specific  survey  and  interview  questions  have  been  generated  for  each  of  the  five 
dimensions  of  collaborative  capacity  presented  in  the  figure  above.  Illustrative  questions  for 
each  of  the  dimensions  are  presented  below: 

Purpose  and  Strategy  questions  address  organizational  purpose,  goals,  and  values; 
the  degree  of  perceived  “felt  need”  to  collaborate;  and  strategic  planning  processes. 

■  Interagency  collaboration  is  a  high  priority  for  this  organization. 

■  We  have  clearly  established  goals  for  interagency  collaboration. 

■  We  consistently  use  an  interagency  approach  to  planning. 

Collaborative  Structure  includes  policies,  roles  and  responsibilities  that  facilitate  or 
serve  as  barriers  to  collaboration;  formal  control  mechanisms  including  authority  and 
standard  operating  procedures;  and  coordinating  structures. 

■  Our  organization  is  flexible  in  adapting  our  procedures  to  better  fit  with  those  of 
partner  organizations. 

■  My  organization  has  mechanisms  in  place  to  monitor  and  evaluate  collaborative 
efforts. 

■  Conflicting  organizational  policies  make  collaboration  very  difficult. 

Social  Capital  through  Lateral  Mechanisms  addresses  both  formal  and  informal 
factors,  including  network  ties,  information  sharing,  combined  training,  and  familiarity  with 
other  organizations.  These  factors,  working  together,  can  become  internalized  into  a  culture 
of  collaboration: 
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Our  organization  has  strong  norms  that  encourage  sharing  information  with  other 
agencies. 


■  Our  organization  commits  adequate  human  and  financial  resources  to  training  with 
other  agencies. 

■  Our  organization  invests  time  and  resources  to  become  familiar  with  the  capabilities 
and  requirements  of  our  partner  organizations. 

Incentives  address  both  the  factors  that  encourage  and  discourage  organizational- 
and  individual-level  engagement  in  collaboration.  The  structure  of  incentives  can  shape 
whether  organizations  frame  their  interactions  as  collaborative  or  competitive. 

■  A  history  of  competition  and  conflict  affects  our  interagency  capability. 

■  Our  organization  rewards  members  for  their  interagency  collaborative  activities. 

■  The  senior  leaders  of  our  organization  often  discuss  the  importance  of  interagency 
collaboration  with  others  in  the  organization. 

People  are  the  foundation  for  macro-level  collaboration,  which  ultimately  depends  on 
their  perceptions,  motives,  attitudes,  and  skills. 

■  Members  of  our  organization  respect  the  expertise  of  those  in  other  organizations 
with  whom  we  have  to  work. 

■  We  have  training  in  place  to  develop  collaborative  skills  (e.g.,  conflict  management, 
team-process  skills). 

■  People  in  our  organization  tend  to  be  suspicious  and  distrustful  of  our  partners  in 
other  organizations. 

We  expect  that  the  ability  to  systematically  assess  collaborative  capacity  can 
contribute  to  something  akin  to  a  common  doctrine  and  common  operational  picture  that  will 
assist  leaders  in  developing  action  plans  for  developing  this  important  capability.  The 
diagnostic  process  encourages  a  common  language  and  understanding  around 
collaboration  and  assists  leaders  in  determining  capabilities  that  the  organization  must 
develop  to  be  successful.  The  next  section  shows  how  our  diagnostic  tool  can  leverage 
learning  for  an  organization. 

Process  for  Diagnosing  Collaborative  Capacity 

We  have  shaped  the  process  for  using  the  collaborative  capacity  diagnostic  around 
the  well-established  principles  of  organization  development  (Beckhard,  1969).  The  focus  in 
this  presentation  is  how  to  use  the  Collaborative  Capacity  survey  instrument  to  inform 
leaders  and  change  agents  of  the  strengths  and  weaknesses  of  their  organization’s 
collaborative  systems.  From  these  data,  specific  interventions  can  be  identified  and 
implemented.  The  survey  tool  is,  thus,  designed  to  contribute  to  a  learning  process  that 
improves  interagency  relationships. 
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This  approach  follows  the  process  of  a  “gap”  analysis  (e.g.,  Harrison,  1994).  In 
consultation  with  the  client  organization(s),  the  diagnostic  process  identifies  the  desired 
future  state — why  collaboration  is  needed  and  ways  in  which  improvements  in  this  capability 
can  be  accomplished.  Through  the  dialogue  that  occurs  in  the  design,  conduct,  and 
analysis  of  survey  results,  organizational  members  become  sensitized  to  the  importance  of 
the  issues  being  assessed  (Downs  &  Adrian,  2004).  The  data  from  the  diagnostic  survey 
also  provide  a  mechanism  to  challenge  existing  mental  models  or  assumptions  of 
organizational  members  about  inter-organizational  collaboration.  They  provide  a  common 
basis  for  understanding  the  “current  state”  and  can,  thus,  motivate  desired  improvements. 

The  key  question  being  addressed  in  interpretation  is,  “What  do  the  assessment 
results  mean?”  In  action  planning,  the  question  is  “What  do  we  do  about  it?”  The 
organizational  members  engaged  in  action  planning  may  be  different  (or  in  addition  to)  those 
who  were  involved  in  the  interpretation.  It  is  important  to  involve  members  in  deciding  what 
action  to  take  if  their  commitment  or  capabilities  are  necessary  to  the  implementation  of  the 
action  plan.  Feedback  about  the  diagnostic  process  should  include  not  only  the  results  and 
interpretation  of  the  assessments,  but  also  the  interventions  identified  as  part  of  action 
planning.  Ongoing  communication  through  the  implementation  of  action  planning  is  also 
important  if  the  diagnostic  process  is  to  contribute  broadly  to  organizational  learning  (Downs 
&  Adrian,  2004;  Senge,  1992). 

The  initial  assessment  establishes  a  baseline  that  can  be  used  to  evaluate  progress 
toward  the  desired  goals  after  the  implementation  of  interventions.  The  assessment  also 
allows  the  opportunity  for  comparisons  across  organizational  levels  and  units.  For  example, 
it  may  be  worthwhile  to  investigate  the  extent  to  which  top-level  managers’  assessment  of 
collaborative  capacity  is  similar  to  those  of  front-line  workers.  Also  of  interest  could  be  a 
comparison  of  those  whose  work  involves  them  with  counterparts  in  other 
organizations/agencies  with  those  who  have  less  frequent  contact. 

Next  steps — ^Validating  the  Assessment  Tool  with  Field  Testing 
and  Subject-matter  Experts  (SMEs) 

There  is  a  growing  body  of  literature  on  the  concept  of  a  capacity  for  collaboration, 
and  our  results  to  date  coincide  with  others  who  are  working  in  this  area  (e.g.,  Foster- 
Fishman,  Berkowitz,  Lounsbury,  Jacobson,  &  Allen,  2001;  Hansen  &  Nohria,  2004;  Huxham, 
1996;  Mankin,  Cohen,  &  Fitzgerald,  2004).  What  we  add  to  those  pursuing  these  ideas 
(e.g.,  Bardach,  1998)  is  a  way  to  measure  the  overarching  concept  of  collaborative  capacity 
and  the  contributing  variables.  Generating  valid  and  reliable  interview  questions  and  survey 
items  require  a  painstaking  process  of  refinement,  testing  and  retesting.  All  questions  are  to 
be  subjected  to  critical  review  by  subject-matter  experts  from  the  acquisition  community  as 
well  as  from  homeland  security  organizations  currently  implementing  or  initiating  inter- 
organizational  collaboration.  The  interagency  level  of  analysis  is  complex  and  requires 
analysis  from  a  variety  of  possible  contexts,  forms,  structures  and  processes.  Our  goal  is  to 
develop  an  audit  that  is  sufficiently  generalizable  to  be  conducted  in  a  wide  variety  of 
contexts,  but  it  must  be  specific  enough  to  provide  actionable  insights  to  organizational 
leaders. 

In  this  stage  of  our  research,  we  are  identifying  potential  partners  who  are  interested 
in  assessing  collaborative  capacity.  These  partners  will  allow  us  to  field  test  the  instrument 
with  organizations  that  are  in  different  developmental  stages.  In  other  words,  some 
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organizations  may  have  only  recently  initiated  the  process  of  collaborating;  others  may  have 
been  collaborating  for  some  time,  but  face  the  problems  of  institutionalizing  and  formalizing 
the  process;  and  some  may  have  institutionalized  their  processes.  Different  lessons  can  be 
learned  in  each  of  these  contexts. 

We  should  also  note  that  the  process  of  validating  items  is  also  a  process  of 
validating  and  elaborating  theoretical  constructs.  This  means  that  the  nuts-and-bolts 
process  of  revising  and  interpreting  items  through  field  testing  itself  generates  more 
coherent  and  useful  ways  of  thinking  about  the  capabilities  and  capacities  of  interagency 
collaboration.  For  example,  we  anticipate  developing  some  preliminary  hypotheses  about 
the  developmental  stages  of  collaborative  capacity  as  we  begin  our  field  testing  work  with 
organizations  that  have  different  amounts  of  experience  with  interagency  collaboration.  We 
also  expect  that  we  will  begin  to  identify  somewhat  of  a  hierarchy  of  predictors  of 
collaborative  capacity  because  it  is  unlikely  that  all  factors  included  in  our  current  model  are 
of  equal  impact  in  influencing  collaboration.  As  we  proceed  with  our  research,  we  will  be 
developing  a  more  refined  diagnostic  process,  as  well  as  a  more  refined  understanding  of 
how  collaborative  capacity  develops  and  ways  it  can  be  fostered. 
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Telecommunication  Systems  Management  from  Naval  Post  Graduate  School,  Monterey,  CA  and  a 
Master  of  Arts  in  National  Security  and  Strategic  Studies  from  the  College  of  Naval  Warfare,  NWC 
Newport,  Rl.  He  attended  the  Defense  Acquisition  University  in  Ft.  Belvoir,  VA,  and  is  certified  Level 
III  in  Program  Management.  Captain  Shannon  is  also  a  National  Security  Management  Fellow  of  the 
Maxwell  School  of  Citizenship  and  Public  Affairs,  Syracuse  University. 

CAPT  Shannon's  operational  experience  includes  service  in  the  2nd,  3rd,  5th,  and  7th  Fleets.  He 
initially  served  in  the  Deck  and  Engineering  departments  onboard  the  USS  SHASTA  (AE-33) 
homeported  in  Concord,  CA.  He  was  also  the  Chief  Engineer  onboard  both  the  USS  PEGASUS 
(PHM-1)  in  Key  West,  FL  and  the  USS  O'BRIEN  (DD-975)  in  San  Diego,  CA.  CAPT  Shannon  sailed 
on  several  different  ships  while  serving  as  Material  Officer  (N4)  for  COMDESRON  31 .  The  Red 
Stallions  of  DESRON  31  relocated  to  Pearl  Harbor,  HI  during  his  assignment  where  they  were  then 
known  as  the  Pacific  Fleet  ASW  Squadron  as  they  tested  and  evaluated  various  tactics  and  ASW 
systems  for  the  Navy.  He  was  the  Executive  Officer  onboard  the  USS  CHANCELLORSVILLE  (CG- 
62)  in  San  Diego,  Ca.  This  ship  was  awarded  the  Spokane  Trophy  for  combat  readiness  during  his 
tour.  CAPT  Shannon's  command  tours  include  the  USS  ESTOCIN  (FFG-15)  and  USS  SAMUEL 
ELIOT  MORISON  (FFG-13)  in  Norfolk,  VA.  While  in  command  of  the  ESTOCIN,  he  deployed  to 
South  America  in  support  of  Operation  UNITAS.  The  ESTOCIN  was  recognized  for  Fleet  Retention 
with  the  Golden  Anchor  Award.  As  the  final  Commanding  Officer  of  SAMUEL  ELIOT  MORISON, 
CAPT  Shannon  coordinated  the  transfer  of  that  ship  to  the  Turkish  Navy. 
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Significant  shore  duties  include  an  instructor  tour  at  the  Surface  Warfare  Officer  School  in  Newport, 

Rl  where  he  taught  tactics  to  prospective  Ship  Department  Heads.  He  served  on  the  Joint  Staff  (C4I 
directorate,  J-6)  where  he  was  an  action  officer  principally  responsible  for  support  of  JROC  activities. 
CART  Shannon  was  selected  as  an  Acquisition  Professional  (AP)  in  2000.  As  an  AP,  he  was  the 
Director  of  Development,  Test,  and  Acceptance  of  the  Evolved  SEASPARROW  Missile  (ESSM). 
During  that  assignment,  the  ESSM  successfully  passed  OPEVAL  and  received  Full  Rate  Production 
acceptance  by  the  Service  Acquisition  Executive  (ASN  RDA).  A  plankowner  in  Program  Executive 
Office,  Integrated  Warfare  Systems  (IWS),  CAPT  Shannon  was  the  first  military  member  assigned  to 
be  the  Project  Manager  for  NIFC-CA  (Naval  Integrated  Fire  Control-Counter  Air).  He  currently  serves 
as  the  Program  Manager  for  Future  Combat  Systems  Open  Architecture  (PEO  IWS  7)  and  is 
responsible  for  carrying  out  the  open  architecture  efforts  among  all  warfare  systems  across  the  entire 
Naval  Enterprise. 

Captain  Shannon  is  married  to  the  former  Lisa  Palmier! .  They  have  two  sons,  Patrick  and  Casey,  and 
one  daughter,  Caroline.  Patrick  is  a  Midshipman  at  the  U.S.  Naval  Academy  ('09). 
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Introduction 

This  research  continues  the  exploration  of  the  use  of  the  modular  open  systems 
approach  (MOSA)  as  a  method  for  implementing  an  evolutionary  acquisition  strategy  in 
Department  of  Defense  (DoD)  programs.  The  background  on  the  initial  DoD  and  Navy 
policy  on  using  a  MOSA  approach  in  defense  acquisition  is  presented,  followed  by  a  review 
of  the  initial  research  findings.  A  discussion  is  then  provided  on  the  Navy’s  method  for 
assessing  its  implementation  of  a  MOSA  approach  in  its  acquisition  programs.  This 
discussion  will  focus  on  the  use  of  the  Naval  Enterprise  Open  Architecture  Assessment  Tool 
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(OAAT).  The  primary  purpose  of  this  continuing  research  is  to  provide  an  analysis  of  the 
results  of  the  OAAT  assessment  of  Navy  acquisition  programs. 


Background  on  MOSA  Policy 

DoD  5000.1  states  that,  “a  modular  open  systems  approach  shall  be  employed 
where  feasible”  (Under  Secretary  of  Defense  (AT&L),  2003,  May  12a;  2003,  May  12b). 
Furthermore,  in  April  2004,  the  USD  (AT&L)  issued  a  memorandum  stating,  “all  programs 
subject  to  milestone  review  shall  brief  their  program’s  MOSA  implementation  status  to  the 
Milestone  Decision  Authority  (MDA)  to  determine  compliance”  (Under  Secretary  of  Defense 
(AT&L),  2004,  April  5). 

Later  that  year,  the  Office  of  the  USD(AT&L),  Director  of  Defense  Systems,  issued 
instructions  for  MOSA  implementation  and  identified  the  Open  System  Joint  Task  Force 
(OSJTF)  as  the  DoD  lead  for  MOSA.  This  memo  also  identified  MOSA  as,  “an  integral  part 
of  the  toolset  that  will  help  DoD  achieve  its  goal  of  providing  the  joint  combat  capabilities 
required  in  the  21®*  century,  including  supporting  and  evolving  these  capabilities  over  their 
total  life-cycle”  (Under  Secretary  of  Defense  (AT&L),  2004,  July  7). 

In  addition,  in  August  2004,  Assistant  Secretary  of  the  Navy  (Research, 

Development  &  Acquisition)  (ASN  (RDA))  issued  a  policy  statement  that  developed  a  single 
Navy-wide  Open  Architecture  to  account  for  Surface,  Air,  Submarine,  041 ,  and  Space 
domain  unique  requirements.  That  memo  also  assigned  PEO  IWS  overall  responsibility  and 
authority  for  directing  the  Navy's  OA  Enterprise  effort.  An  OA  Enterprise  Team  comprised  of 
OA  domain  leads,  ASN,  OPNAV,  and  SYSCOM  representatives  was  chartered  and  led  by 
PEO  IWS.  The  Team  collectively  oversees  the  development  and  implementation  of  the 
processes,  business  strategies,  and  technical  solutions  which  support  cross-Enterprise 
requirements  in  addition  to  domain-specific  needs.  The  Enterprise  Team  will  also  define  an 
overarching  OA  acquisition  strategy  and  develop  guidance  that  addresses  incentives, 
intellectual  property  issues,  contracting  strategies  (i.e.,  integrator's  vs.  prime's),  and  funding 
alternatives  (ASN  (RD&A),  2004). 

Finally,  in  a  23  December  2005  letter.  Deputy  Chief  of  Naval  Operations  (Warfare 
Requirements  and  Program)  established  the  Navy-wide  requirement  for  OA  and  laid  out  the 
priorities  on  which  it  wants  Naval  OA  to  focus.  The  letter,  “establishes  the  requirement  to 
implement  Open  Architecture  (OA)  principles  across  the  Navy  Enterprise.”  It  establishes  the 
OA  Council  (OAC)  of  representatives  of  N6/N7  Division 

Directors  to  work  with  the  OAET  on  the  requirements.  The  letter  directs  the  OAC, 
PEO  IWS  7.0,  and  the  OAET  to  focus  assessment  priorities  in  support  of  the  following 
capabilities:  Track  management.  Combat  ID  (CID),  Data  fusion.  Time-critical  Targeting  & 
Strike,  and  Integrated  Fire  Control  (IFC). 

Initial  Research  Findings 

The  purpose  of  the  initial  MOSA  research  was  to  explore  both  the  use  of  the  modular 
open  systems  approach  (MOSA)  as  a  method  for  implementing  an  evolutionary  acquisition 
strategy,  as  well  as  the  implications  of  using  such  an  approach  on  the  contracting  process. 

Although  the  phases  of  the  contracting  process  are  the  same  for  MOSA-based 
programs  as  they  are  for  non-MOSA-based  programs,  this  research  found  that  the  specific 
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activities  conducted  and  documents  deveioped  during  the  execution  of  these  contracting 
phases  have  a  direct  influence  on  the  success  of  a  MOSA-based  program.  For  exampie, 
the  various  options  for  aiiocating  roles  and  responsibiiities  between  the  government  and  the 
contractor  for  the  various  steps  in  the  acquisition  process  will  influence  the  amount  of 
“openness”  in  the  program  and  the  contractor’s  motivation  for  meeting  the  desired  level  of 
openness. 

This  research  indicated  that  the  greater  degree  of  jointness  in  acquisition  roles  and 
responsibilities,  as  well  as  the  greater  degree  of  contractor-developed  acquisition 
documents,  will  lead  to  a  higher  level  of  openness. 

This  initial  research  also  identified  early  involvement  and  participation  by  industry  in 
developing  requirements  and  acquisition  strategy  as  a  key  factor  in  successful  MOSA-based 
programs.  Program  offices  managing  a  MOSA-based  program  should  conduct  extensive 
market  research  and  industry  conferences  to  achieve  this  contractor  involvement.  A  best- 
value  contract  strategy  that  is  tailored  to  emphasize  technical  performance  in  open-based 
systems  and  COTS  systems  is  also  a  critical  factor  in  meeting  higher  levels  of  openness  in 
MOSA-based  programs.  A  contract  strategy  which  involves  developing  source-selection 
evaluation  factors  specifically  weighted  to  emphasize  an  open  systems  approach  will  be 
critical  for  MOSA-based  programs. 

As  important  as  the  acquisition  strategy  is  the  structure  of  the  contract  of  a  MOSA- 
based  program. 

This  research  identified  the  use  of  incentive-fees,  award-fees,  and  award-term 
contract  incentives  as  integral  to  the  success  of  MOSA-based  programs.  These  incentives, 
if  structured  appropriately,  are  effective  tools  for  motivating  and  incentivizing  contractors  to 
achieve  higher  levels  of  openness  in  the  design  and  development  of  systems. 

Finally,  the  consistent  and  aggressive  use  of  the  contractor  past-performance 
information  system,  as  well  as  the  development  and  establishment  of  lessons-learned 
programs  and  best  practices  will  be  essential  as  more  and  more  MOSA-based  programs  are 
initiated.  As  contractors  performing  work  on  MOSA-based  programs  begin  to  realize  that 
the  DoD  is  insistent  on  using  open  systems  in  developing  its  major  weapon  systems,  they 
should  begin  to  dedicate  the  required  resources  to  this  method  of  developing  weapon 
systems. 

Internal  Assessment  of  MOSA  Implementation 

The  focus  of  this  follow-on  research  is  to  analyze  the  effectiveness  of  the 
implementation  of  MOSA  in  Navy  acquisition  programs  by  investigating  the  results  of 
MOSA-internal  assessments,  specifically  the  results  of  the  Open  Architecture  Assessment 
Tool  (OAAT).  The  results  of  this  research  will  prove  beneficial  to  senior  Navy  officials  by 
providing  data  points  on  MOSA  implementation  by  analyzing  the  consistency  of  MOSA 
compliance  status  and  internal  assessments  for  specific  Navy  acquisition  programs. 

The  OAAT  is  a  tool  designed  to  assist  Navy  program  managers  in  assessing  the 
"openness"  of  their  programs.  It  aligns  to  the  Open  Architecture  Assessment  Model  (OAAM) 
as  approved  by  ASN(RDA)  and  provides  a  reproducible  and  objective  method  of  conducting 
program  assessments.  Specifically,  the  OAAT  is  an  analytic  tool  that  evaluates  responses 
to  a  set  of  interrelated  questions  to  provide  program  officers  with  an  objective  and  evidence- 
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based  assessment  of  the  degree  that  a  program  exhibits  openness  along  two  axes: 
business  and  technical.  The  degree  that  openness  is  implemented  is  presented  in  terms  of 
business/programmatic  and  technical  criteria.  The  business/programmatic  dimension 
criteria  include  questions  that  address:  Open  Architecture,  Modular  Open  Design,  Interface 
Design  and  Management,  Treatment  of  Proprietary  Elements,  Open  Business  Practices, 
Peer  Review  Rights,  and  Technology  Insertion.  The  technical  dimension  criteria  cover 
essential  OA  design  tenets  of  Interoperability,  Composability,  Reusability,  Maintainability 
and  Extensibility. 

The  OAAT  assessment  score  summary  provides  a  summary  of  the  ratings  for  each 
of  the  evaluated  areas  (See  Figure  1). 


Business  Areas 

Open  Systems  Approach 
Open  Architecture 
Open  Moduiar  Design 
interface  Design  and  Management 
Treatment  of  Proprietary  Eiements 
Open  Business  Practices 
Peer  Review  Rights 
Technicai  Insertion 
Commercial  Standards 
Compliance 


Technical  Areas 

Design  Tenet:  Interoperability 
Design  Tenet:  Maintainability 
Design  Tenet:  Extensibility 
Design  Tenet:  Composability 
Design  Tenet:  Reusability 
General  Design  Tenet 


Figure  1.  Ratings  of  Evaluated  Areas 

In  addition,  an  OA  assessment  matrix  that  displays  the  program  current  state  with 
respect  to  business  and  technical  openness  is  also  provided  in  the  assessment  summary. 
Each  of  these  areas  (business  and  technical)  is  rated  on  a  scale  of  0  to  4.  (See  Figure  2.) 
The  results  of  the  OAAT  assessment  could  then  be  used  by  the  program  manager  to  help 
improve  the  program  with  respect  to  Naval  Open  Architecture. 
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Figure  2.  Open  Architeture  Maturity  Matrix 


This  is  an  executive  summary  of  the  complete  research  report.  The  complete  research 

report  may  be  accessed  from  the  Naval  Postgraduate  School  website 

www.nps.navv.mil/qsbpp/acan/publications. 
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“Software  architecture  forms  the  backbone  for  any  successful  software-intensive 
system.  An  architecture  is  the  primary  carrier  of  a  software  system’s  quality 
attributes  such  as  performance  or  reliability.  The  right  architecture — correctly 
designed  to  meet  its  quality  requirements,  clearly  documented,  and  conscientiously 
evaluated — is  the  linchpin  for  software  project  success.  The  wrong  one  is  a  recipe 
for  guaranteed  disaster”  (Software  Engineering  Institute/Carnegie  Mellon,  2007). 

Introduction 

Software  engineers  will  typically  spend  50%  or  more  of  the  total  software 
development  time  designing  software  architecture,  and  that  architecture  may  provide  up  to 
80%  of  a  modern  weapon  system’s  functionality.  Increasingly,  these  systems  will  operate 
within  a  network  or  other  system-of-systems  architecture.  Obviously,  the  requirements 
driving  that  architectural  design  effort  and  the  process  for  tracing  requirement  to  functions, 
insight  into  the  process,  and  control  of  the  effort  are  critical  for  the  successful  development 
of  the  capability  needed  by  the  warfighter. 
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The  DoD  typically  monitors  and  controls  system  technical  development  through 
implementation  of  the  Baselines,  Audits  and  Technical  Reviews  within  an  overarching 
Systems  Engineering  Process  (SEP)  (Defense  Acquisition  University,  2004,  December, 
chap.  4).  Because  of  the  relatively  immature  software  engineering  environment,  significantly 
more  analysis  and  development  of  the  requirements  is  required.  In  addition,  the  software 
architectural  design  effort  is  dependent  on  in-depth  requirements  analysis,  is  resource 
intensive,  and  must  occur  very  early  in  the  process.  Effective  management  and 
implementation  of  design  metrics  is  essential  in  developing  software  that  meets  the 
warfighters’  needs.  This  management  and  metrics  effort  supplements  and  supports  the 
system  technical  development  through  the  Baselines,  Audits  and  Technical  Reviews. 

There  are  numerous  variations  and  models  of  the  Systems  Engineering  Process 
(SEP).  This  research  uses  the  model  depicted  in  Figure  1  (below),  which  illustrates  the 
systems  engineering  functions  described  throughout  this  paper.  The  concepts  are 
transferable  to  the  SEP  “V”  model  currently  used  by  the  DoD. 


Figure  1.  Systems  Engineering  Process 

Software  Requirements  Impact 

The  importance  of  system  software  requirements  development  to  the  potential 
success  of  software-intensive  systems  development  cannot  be  overstated. 
Underdeveloped,  vaguely  articulated,  ill-defined  software  requirements  elicitation  has  been 
linked  to  poor  cost  and  schedule  estimations — resulting  in  disastrous  cost  and  schedule 
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overruns.  In  addition,  the  resulting  products  have  been  lacking  important  functionality,  are 
unreliable,  and  have  been  costly  and  difficult  to  effectively  sustain  (Naegle,  2006, 
September). 

Using  the  SEP  approach,  the  explicit  user  capabilities  requirements  specified  in  the 
Joint  Capabilities  Integration  and  Development  System  (JCIDS)  provides  the  Input  for 
system  Requirements  Analyses.  These  analyses  are  intended  to  illuminate  all  system- 
stated,  derived  and  implied  requirements  and  quality  attributes  necessary  to  achieve  the 
capabilities  needed  by  the  warfighter.  The  Work  Breakdown  Structure  (WBS)  is  a 
methodology  for  defining  ever-increasing  levels  of  performance  specificity — using  the  SEP 
to  guide  the  development  of  each  successive  layer  (Department  of  Defense,  2005,  July,  pp. 
1-5). 

Software  Engineering  Environment 

The  software  engineering  environment  is  not  mature,  especially  when  compared  to 
hardware-centric  engineering  environments.  Dr.  Philippe  Kruchten  of  the  University  of 
British  Columbia  remarks,  “we  haven’t  found  the  fundamental  laws  of  software  that  would 
play  the  role  that  the  fundamental  laws  of  physics  play  for  other  engineering  disciplines” 
(Kruchten,  2005,  p.  17).  Software  engineering  is  significantly  unbounded  as  there  are  no 
physical  laws  that  help  define  environments;  and  to  date,  no  industry-wide  dominant 
language,  set  of  engineering  tools,  techniques,  reusable  assets,  or  processes  have 
emerged. 

This  lack  of  engineering  maturity  impacts  both  requirements  development  and  the 
subject  for  this  research,  design  of  the  architecture,  which  will  be  discussed  later.  To 
compensate  for  the  relative  immaturity  of  the  software  engineering  environment,  the  DoD 
must  conduct  significantly  more  in-depth  requirements  analysis  and  provide  potential 
software  developers  detailed  performance  specifications  in  all  areas  of  software 
performance  and  sustainability. 

Performance  Specifications  and  the  Work  Breakdown  Structure 

(WBS) 

Since  the  implementation  of  Acquisition  Reform  in  the  nineties,  detailed 
specifications  have  been  replaced  with  performance  specifications  in  order  to  leverage  the 
considerable  experience  and  expertise  available  in  the  defense  contractor  base.  In  most 
hardware-centric  engineering  disciplines,  the  expertise  the  DoD  seeks  to  leverage  includes 
a  mature  engineering  environment  in  which  materials,  standards,  tools,  techniques  and 
processes  are  widely  accepted  and  implemented  by  industry  leaders.  This  engineering 
maturity  helps  to  account  for  derived  and  implied  requirements  not  explicitly  stated  in  the 
performance  specification.  Three  levels  of  the  WBS  may  provide  sufficient  detail  for  a 
desired  system  to  be  developed  in  a  mature  engineering  environment,  such  as  the 
automotive  field.  For  example,  an  automotive  design  that  provides  for  easy  replacement  of 
wear-out  items  such  as  tires,  filters,  belts,  and  batteries  obviously  provides  sustainability 
performance  that  is  absolutely  required.  Most  performance  specifications  do  not  explicitly 
address  this  capability  as  they  would  be  automatically  considered  by  any  competent 
provider  within  the  mature  automotive  engineering  environment. 

In  stark  comparison,  the  software  engineering  environment  offers  little  assistance  in 
compensating  for  derived  and  implied  requirements,  and  developers  are  limited  to  respond. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  -  269  - 


almost  exclusively,  to  the  explicit  requirements  provided.  The  DoD  Handbook  881  A,  “Work 
Breakdown  Structures  for  Defense  Materiel  Items,”  recommends  a  minimum  of  three  levels 
be  developed  before  handoff  to  a  contractor.  If  a  program  is  expected  to  be  high-cost  or 
high-risk,  it  is  critical  to  define  the  system  at  a  lower  level  of  the  WBS  (Department  of 
Defense,  2005,  July,  p.  3).  Complex  weapon  systems  are  nearly  always  high-cost,  and  the 
complex  software  development  needed  almost  always  means  that  It  is  high-risk,  as  well. 

The  WBS  and  performance  specification  must,  consequently,  be  significantly  more 
developed  to  provide  the  software  engineer  enough  information  and  insight  to  accurately 
estimate  the  level  of  effort  needed — cost  and  schedule — and  to  actually  produce  the 
capabilities  needed  by  the  warfighter.  Contracts  resulting  from  proposals  that  are  based  on 
underdeveloped,  vague,  or  missing  requirements  typically  result  in  catastrophic  cost  and 
schedule  growth  as  the  true  level  of  software  development  effort  is  discovered  only  after 
contract  award. 

The  WBS  provides  the  basis  for  the  performance  specification  and  is  a  powerful 
communications  medium  with  potential  contractors  as  the  upper  levels  provide  a  functional 
system  breakdown  structure  from  the  DoD’s  perspective.  The  same  WBS  continues  to  be 
developed  by  the  contractor,  eventually  providing  the  detailed  breakdown  structure,  which  is 
the  basis  for  the  cost  and  scheduling  estimates  provided  in  the  proposals  and  used  in  the 
Earned  Value  Management  (EVM)  metrics  during  execution. 

Software  Quality  Attributes 

As  the  system  requirements  are  developed,  software  quality  attributes  are  identified 
and  become  the  basis  for  designing  the  software  architecture.  One  methodology  for  fully 
developing  the  software  attributes  is  to  use  the  Software  Engineering  Institute’s  Quality 
Attribute  Workshop  (QAW),  which  is  implemented  before  the  software  architecture  has  been 
created  and  is  intended  to  provide  stakeholder  input  about  their  needs  and  expectations 
from  the  software  (Barbacci,  Ellison,  Lattanze,  Stafford,  Weinstock,  &  Wood,  2003,  August, 


p.  1). 


While  the  QAW  would  certainly  be  useful  after  contract  award,  conducting  the 
workshop  between  combat  developers/users  and  the  program  management  office  before 
issuance  of  the  Request  for  Proposal  (RFP)  would  provide  an  improved  understanding  of 
the  requirements,  enhance  the  performance-specification  preparation,  and  improve  the 
ability  of  the  prospective  contractors  to  accurately  propose  the  cost  and  schedule.  This 
approach  would  support  the  goals  of  the  System  Requirements  Review  (SRR),  which  is 
designed  to  ascertain  whether  all  derived  and  implied  requirements  have  been  defined. 

The  QAW  process  provides  a  vehicle  for  keeping  the  combat  developer  and  user 
community  involved  in  the  DoD  acquisition  process,  which  is  a  key  goal  of  that  process.  In 
addition,  the  QAW  includes  scenario-building  processes  that  are  essential  for  the  software 
developer  In  designing  the  software  system  architecture  (Barbacci,  Ellison,  Lattanze, 
Stafford,  Weinstock,  &  Wood,  2003,  August,  pp.  9-11).  These  scenarios  will  continue  to  be 
developed  and  prioritized  after  contract  award  to  provide  context  to  the  quality  attribute. 
Specific  recommendations  for  this  process  will  be  discussed  later. 

Maintainability,  Upgradability,  Interoperability/Interfaces,  Reliability, 
and  Safety/Security  (MUIRS)  Analytic  Technique 

The  QAW  provides  the  “how,”  and  the  performance  requirements  (with 
Maintainability,  Upgradability,  Interoperability/Interfaces,  Reliability,  and  Safety/Security 
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(MUIRS)  analytic  technique)  provides  the  “what” — or  at  least  a  significant  portion  of  it.  The 
MUIRS  elements  also  help  capture  the  need  for  Open  Architecture  (OA),  especially  in  the 
Maintainability,  Upgradability,  and  Interoperability/Interfaces  elements.  Much  of  the 
software  performance  that  typically  lacks  consideration  and  is  not  routinely  addressed  in  the 
software  engineering  environment  can  be  captured  through  development  and  analysis  of  the 
MUIRS  elements.  Analyzing  the  warfighter  requirements  in  a  QAW  framework  for 
performance  in  each  MUIRS  area  will  help  identify  software  quality  attributes  that  need  to  be 
communicated  to  potential  software  contractors  (Naegle,  2006,  September,  pp.  17-24). 
While  this  technique  would  be  effective  within  any  system,  it  is  especially  effective  in 
compensating  for  the  lack  of  software  engineering  maturity  and  in  conveying  a  more 
complete  understanding  of  the  potential  software-development  effort,  resulting  in  more 
realistic  proposals. 

The  MUIRS  analytical  approach  provides  a  framework  to  capture,  develop,  and 
document  derived  and  implied  requirements — ^which  may  be  vaguely  alluded  to  or  missing 
from  the  user/combat  developer’s  requirements  documents.  For  example,  a  user 
requirement  might  be  simply  presented  in  terms  like,  “The  network  must  be  secure  in  all 
modes  within  the  intended  environment.”  Without  further  development,  the  software 
engineer  may  interpret  that  requirement  in  many  different  ways,  planning  for  authentication 
and  encryption/decryption  routines.  Applying  the  Safety/Security  analytic  approach  in  a 
QAW  format,  the  derived  and  implied  requirements  are  likely  to  elucidate  the  following 
requirements: 

■  Ability  to  constantly  monitor  the  network  to  detect  and  counteract  active  or  passive 
intrusion  or  attacks 

■  Ability  to  provide  details  of  attacks  to  Intelligence/Counter  Intelligence  personnel 

■  Ability  to  conduct  passive  measures  to  ensure  that  all  node  operations  are  conducted 
with  authorized  personnel  exclusively 

■  Ability  to  quarantine  a  suspect  node  without  impacting  the  rest  of  the  network.  Ability 
to  lift  the  quarantine  when  properly  authenticated. 

■  Ability  to  identify  information  provided  to,  or  requested  by  the  quarantined  node  for 
Intelligence/Counter  Intelligence  analysis 

■  Passive  ability  to  authenticate  information  sources 

■  Ability  to  interoperate  with  other  secure  devices  and  networks  without  the  risk  of 
compromise 

■  Ability  to  accommodate  network  system  changes  and  upgrades 

■  Ability  to  accommodate  a  wide  array  of  users  and  organizations,  formed  into  the 
secure  network  task  force  as  missions  dictate 

The  difference  in  the  level  of  requirement  development  is  significant,  and  the  more 
complete  information  provides  necessary  performance  thresholds  that  must  be 
accommodated  by  the  software  design  and  development  effort.  The  software  architecture 
would  likely  be  vastly  different  the  implied  and  derived  security  requirements  are  considered. 
The  amount  of  work  required  to  meet  the  actual  software  security-performance  attributes  is 
revealed  to  the  contractor  prior  to  proposal  preparation — which  should  vastly  improve  the 
cost  and  schedule  accuracy  of  the  proposal  submitted.  In  addition,  the  software  engineer 
gains  a  much  more  in-depth  understanding  of  the  system  being  developed,  thereby 
improving  the  design  effort  described  later. 
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Similar  analyses  of  all  MUIRS  elements  provide  a  much  more  complete 
understanding  of  requirements  and  insight  into  the  operational  environment  envisioned  by 
the  warfighter.  This  level  of  understanding  is  absolutely  crucial  for  effective  design  of  the 
software  architecture.  If  the  design  effort  is  started  without  this  level  of  understanding  of  the 
requirement  attributes,  significant  architectural  design  rework  or  outright  scrapping  of  early 
design  attempts  is  inevitable — resulting  in  increased  costs  and  lengthened  schedules. 


Software  Architecture  Characteristics 

Software  Developer  Effort 

In  past  acquisition  programs,  software  development  was  considered  something  that 
could  be  fielded  and  then  “fixed”  after  the  weapon  systems  were  deployed.  The  complexity 
of  software,  interface  problems  and  the  cost  for  rework  were  grossly  underestimated;  the 
result  was  costly  schedule  slips  and  less-than-desired  performance. 

When  software  development  was  in  its  infancy  in  1968,  Alfred  M.  Peitrasanta  at  IBM 
Systems  Research  Institute  wrote: 

Anyone  who  expects  a  quick  and  easy  solution  to  the  multi-faced  problem  of 
resource  estimation  is  going  to  be  disappointed.  The  reason  is  clear;  computer 
program  system  development  is  a  complex  process;  the  process  itself  is  poorly 
understood  by  its  practitioners;  the  phases  and  functions  which  comprise  the  process 
are  influenced  by  dozens  of  ill-defined  variables;  most  of  the  activities  within  the 
process  are  still  primarily  human  rather  than  mechanical,  and  therefore  prone  to  all 
the  subjective  factors  which  affect  human  performance.  (Pietrasanta,  1968,  pp.  341- 
346) 

After  numerous,  costly  software  disasters,  we  have  learned  that  software 
development  must  be  a  parallel  effort  with  system  development  within  the  acquisition 
framework  to  ensure  that  requirements  are  being  met  and  usable  products  are  being 
delivered  to  the  warfighter.  As  the  system  requirements  are  defined,  the  requirements  for 
the  software  should  also  be  developed  concurrently.  One  critical  factor  in  the  software 
development  effort  is  applying  systems  engineering  discipline  to  the  process  and  ensuring 
that  discipline  is  continuous  and  rigorous  throughout  the  development.  Software 
development  has  the  highest  degree  of  program  risk  and  tends  to  evolve  into  a  state  of 
turmoil,  which  is  detrimental  to  the  goal  of  mission-ready  software  and  has  a  negative  impact 
on  cost,  schedule  and  performance. 

Software  Functionality  and  Design  Architecture 

The  design  of  the  architecture  begins  with  the  description  of  the  system  and 
identifies  the  functions  required  for  the  system  to  provide  the  capabilities  desired.  The 
required  functions  will  drive  the  design  of  the  system  architecture.  System  functionality  and 
performance  requirements  are  documented  in  the  Government’s  Request  for  Proposal 
(RFP).  The  potential  contractor  must  break  down  those  functions  and  performance 
requirements  and  consider  Maintainability,  Upgradeability,  Interfaces/Interoperability, 
Reliability,  Safety,  and  Security  (MUIRSS)  in  the  design-decision  process.  The  MUIRSS 
analysis  will  ensure  the  contractor  understands  the  requirement  and  will  also  identify  any 
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limiting  factors  in  the  system  requirements  tradeoffs.  The  desired  functionality  and  the 
analysis  will  drive  the  system  architecture.  For  software-intensive  acquisition  programs,  it  is 
even  more  critical  that  the  performance  requirements  be  communicated  and  understood  by 
the  software  developer. 


Work  Breakdown  Structure 

The  Government’s  requirements  and  specifications  for  a  new  weapon  system  are 
detailed  in  the  RFP;  this  includes  a  Government-produced  Work  Breakdown  Structure 
(WBS)  to  at  least  three  levels.  This  is  known  as  the  Program  WBS  and  is  handed  off  to  the 
contractor  to  develop  a  WBS  that  defines  the  level  of  detail  required  for  product 
development.  This  contractor-generated  product  will  ensure  the  system  developer 
understands  the  program  objectives  and  the  products  to  be  delivered  in  performance  of  the 
contract.  The  WBS  details  the  functionality  and  performance  of  the  system  and  provides  a 
baseline  to  track  performance  against  cost  and  schedule.  For  most  hardware-centric 
programs,  a  WBS  for  the  top  three  levels  of  the  system  under  development  is  usually 
enough  detail  to  manage  the  program.  Because  of  the  volatile  nature  of  software 
development,  immature  software  engineering  environment,  and  the  potential  impact  to  cost, 
schedule  and  risk,  the  WBS  for  software  intensive  programs  need  to  be  developed  down  to 
Level  5  or  lower  for  a  software-intensive  program — including  system-of-systems  (SOS)  and 
net-centric  systems  development. 

Level  1  of  the  WBS  describes  the  entire  project.  If  the  program  is  a  Systems  of 
Systems  (SOS)  project.  Level  I  becomes  that  overarching  system.  The  Army  Future 
Combat  System  (FCS)  has  a  number  of  platforms  that  are  segments  of  the  total  system. 
Each  platform  becomes  a  major  segment  of  that  product  (Level  2);  the  software 
development  would  then  be  broken  down  to  Level  6,  which  identifies  software-configuration 
items. 

Using  the  FCS  as  an  example.  Level  1  describes  the  overall  FCS  concept  and 
environment.  Level  2  details  the  major  product  segments  of  the  SOS.  With  our  example  of 
FCS,  the  Level  2  would  be  the  manned  systems,  i.e.,  infantry-carrier  vehicles,  command- 
and-controlled  vehicles,  mounted  combat  systems,  etc. 

Level  3  defines  the  major  components  or  subsets  of  Level  2.  For  software 
development,  decomposing  the  software  WBS  to  the  lowest  component  is  critical  for  the 
developer  to  fully  comprehend  the  detailed  level  of  effort  required  to  design  and  develop 
effective  systems.  Under  the  FCS  scenario.  Level  3  would  be  one  of  the  subsystems  on 
board  the  manned  systems,  e.g.,  the  fire-control  systems  and  environmental-control 
systems.  It  is  clear  that  WBS  definition  to  this  level  provides  only  a  very  top-level  insight  to 
the  system  being  developed;  thus,  for  the  software-intensive  system,  the  WBS  fails  to 
convey  enough  information  for  the  contractor  to  propose  a  realistic  cost  and  schedule 
estimate.  Too  much  of  the  software  development  work  is  hidden  at  this  level. 

Level  4  becomes  a  breakout  of  the  component  parts  of  the  subsystem.  Using  a 
manned  vehicle  in  the  FCS  program.  Level  5  of  the  WBS  would  identify  the  component 
functions  for  the  fire-control  system:  for  example,  detect  the  target,  aim  at  the  target  and  fire 
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the  munitions.  The  software  build  set  would  support  the  functionality  of  that  component 
within  the  subsystem.  Again,  using  FCS  as  the  overarching  program,  Level  6  is  a  sum  of 
software  items  (Si’s)  which  satisfy  a  required  function  and  are  designated  for  configuration 
management.  If  the  software  requirements  or  attributes  are  well  defined,  the  result  is  a 
product  that  is  properly  designed  to  functionally  perform  to  the  users’  requirements.  Further 
development  below  Level  6  may  be  necessary  to  adequately  convey  the  derived  and  implied 
requirements  needed  by  the  software  developer. 

Systems  Engineering  Process 

Just  as  it  supports  hardware  development,  the  Systems  Engineering  Process  (SEP) 
is  essential  in  the  development  of  software  design.  In  software  development,  good  quality 
and  predictable  results  are  paramount  goals  in  creating  the  specified  warfighter  capabilities 
within  cost  and  schedule  constraints.  To  accomplish  those  goals,  we  examine  the  methods, 
tools  and  processes  that  the  software  developer  uses  in  building  the  software  with  the  intent 
of  attaining  a  product  that  provides  all  of  the  necessary  functionality  and  is  supportable, 
efficient,  reliable  and  easy  to  upgrade. 

The  SEP  also  helps  identify  and  manage  program  risk.  How  mature  is  the  processes 
of  the  software  developer?  One  cause  for  delays  and  cost  overruns  in  the  C-17 
Globemaster  program  was  the  contractor’s  lack  of  software  experience,  which  is  an  element 
of  the  developer’s  maturity.  To  address  developer  maturity,  SEI  developed  an  evaluation 
tool  in  the  mid-1980s  known  as  the  Capability  Maturity  Model  (CMM)  which  rates  software 
developers  on  key  elements  of  maturity  including  experience,  processes,  management  and 
demonstrated  predictability.  This  gives  the  DoD  insight  into  the  maturity  of  potential 
developers  as  a  means  of  risk  reduction. 

The  system  requirements,  stated  in  the  RFP,  detail  the  software’s  functions,  what  it 
must  do  and  how  well,  under  what  conditions,  and  identifies  interfaces  and  interoperability 
requirements.  The  performance  requirements  are  also  analyzed  for  required  response  time, 
maintainability  and  modularity,  open-architecture  requirements  and  transportability.  This 
phase  of  the  SEP  also  addresses  any  restricting  factors — ^for  example,  interface  with  legacy 
systems,  any  required  operating  systems — and  also  identifies  issues  such  as  data  and 
software  rights  constraints. 

The  developer  then  identifies  software  attributes  and  decomposes  functions  to  the 
lowest  level,  ensuring  that  each  performance  specification  in  the  RFP  has,  as  a  minimum, 
one  function.  The  functional  architecture,  the  block  diagrams  and  software  interfaces  are 
described  during  this  step. 

These  functions  are  then  combined  into  a  system  that  describes  the  architecture, 
defines  all  interfaces,  explains  operating  parameters,  produces  the  Si’s  and  develops  the 
documentation,  technical  manuals,  and  any  deliverable  media  (Kazman,  Klein  &  Clements, 
2000,  August,  p.  vii). 

Attribute-driven  Design 

“Quality  attribute  goals,  by  themselves,  are  not  definitive  enough  either  for  design  or  for 
evaluation”  {Barbacc\,  Ellison,  Lattanze,  Stafford,  Weinstock,  &  Wood,  2003,  August,  p.  3) 
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The  design  of  the  system  architecture  will  be  driven  by  the  quality  attributes 
requirements.  The  performance  goals  of  the  system  must  be  defined  not  only  in  attributes  or 
qualities,  but  also  in  how  those  attributes  interact  or  interface  with  the  system  and 
subsystems.  If  those  attributes  are  poorly  communicated,  the  architectural  design  will  fail  to 
meet  the  performance  goals  and  could  potentially  impact  the  overall  program  cost  and 
schedule.  Those  critical  attributes  or  qualities  must  be  carefully  documented  and  articulated 
to  the  software  designer.  To  evaluate  the  architecture,  the  designer  must  receive  a  detailed 
description  of  the  desired  attributes  with  the  overall  proposed  design  of  the  system. 

However,  in  the  evaluation  of  the  design,  an  analysis  of  the  attributes  may  not  be  enough 
detail  for  the  developer.  The  RFP  or  performance  specification  needs  to  address  any 
operational  requirements  or  constraints.  Clearly,  understanding  the  attributes  in  the  context 
of  how  they  are  used  is  critical  for  the  software  designer. 

Software  Architecture  Analysis 

If  a  software  architecture  is  a  key  business  asset  for  an  organization,  the 
architectural  analysis  must  also  be  a  key  practice  for  that  organization.  Why? 
Because  architectures  are  complex  and  involve  many  design  tradeoffs.  Without 
undertaking  a  formal  analysis  process,  the  organization  cannot  ensure  that  the 
architectural  decisions  made — particularly  those  which  affect  the  achievement  of 
quality  attributes  such  as  performance,  availability,  security,  and  modifiability — are 
advisable  ones  that  appropriately  mitigate  risks.  (Kazman,  Klein  &  Clements,  2000, 
August,  p.  vll) 

This  quote  from  the  Software  Engineering  Institute  illustrates  the  importance  of 
performing  architectural  analysis  in  developing  software-intensive  systems. 

After  thorough  requirements  development  and  elicitation,  architectural  analysis  Is  the 
next  necessary  step  in  managing  the  software  development  and  serves  as  the  SEP 
functional  allocation  step.  Defining  the  requirements  and  software  quality  attributes  is  a 
critical  first  step  to  any  program  development  and  provides  the  basis  for  architectural 
analysis.  One  of  the  main  functions  of  the  architectural  analyses  is  to  understand  how  the 
quality  attribute  is  being  achieved  by  the  design  architecture  and,  just  as  importantly,  is  to 
gain  insight  into  how  those  attributes  interact  with  each  other.  For  example,  it  is  crucial  to 
understand  how  security  is  ensured  while  the  open-system  architecture  the  DoD  requires  is 
maintained. 

Understanding  Quality  Attributes  in  Context 

It  Is  not  sufficient  to  understand  a  quality  attribute  without  understanding  the  context 
in  which  it  will  be  used  and  sustained  by  the  warfighter.  One  method  of  gaining  the  needed 
context  is  to  develop  operational  scenarios  that  would  place  all  software  quality  attributes 
into  system-use  cases  spanning  key  effectiveness  and  suitability  issues.  The  development 
and  prioritization  of  the  operational  scenarios  must  be  accomplished  by  the  user,  combat 
developer,  warfighter,  and  other  stakeholders — keeping  them  actively  engaged  in  the 
developmental  process. 

The  context  in  which  the  attributes  function  provides  significant  design  cues  to  the 
software  engineer.  For  example,  the  M1 A2  Abrams  main  battle  tank  uses  numerous  inputs 
for  precisely  engaging  threat  targets.  Several  such  inputs  are  essential  for  any  acceptable 
probability  of  hitting  the  desired  target,  including  target  acquisition  (finding  the  target). 
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location  (azimuth  and  range),  aiming/tracking,  and  firing  the  projectile.  To  increase 
accuracy,  several  other  systems  are  employed  that  enhance  one  or  more  of  the  essential 
functions,  including  cross-wind  sensor,  temperature  sensor,  muzzle-reference  system,  and 
others.  The  tank  main-gun  engagement  scenario  separates  the  essential  functions  from  the 
enhancing  functions,  allowing  the  software  engineer  to  design  the  software  to  permit  an 
engagement  when  all  of  the  essential  functions  are  operational — even  when  an  enhancing 
function,  like  the  temperature  sensor,  is  not  working.  The  warfighter  can  continue  to  fight 
effectively  using  the  system,  increasing  mission  reliability.  Without  development  of  these 
scenarios,  every  requirement  and  quality  attribute  appear  to  be  in  the  “essential”  category, 
which  may  result  In  a  design  that  precludes  critical  operations  when  a  non-essential 
enhancing  system  is  networking. 

Operational  Scenario  Development 

A  scenario  is  a  short  statement  describing  an  interaction  of  one  of  the  stakeholders 
with  the  system  (Kazman,  Klein  &  Clements,  2000,  August,  p.  13).  A  warfighter  would 
describe  using  the  system  to  perform  a  task  or  mission  in  a  range  of  environments  (dark, 
cold  hot,  contaminated,  etc.).  A  leader  would  describe  system  employment  in  concert  with 
other  joint  and  allied  systems  in  a  system-of-systems  approach.  A  system  maintainer  would 
describe  preventative  or  restorative  maintenance  tasks  and  procedures.  A  trainer  would 
describe  programs  of  instruction  to  task,  condition  and  standard. 

Much  of  the  necessary  operational  scenario  development  work  has  been 
accomplished  through  implementation  of  the  Joint  Capabilities  Integration  and  Development 
System  (JCIDS)  (Chairman  of  the  Joint  Chiefs  of  Staff,  2005,  May).  JCIDS  is  the  user’s 
capability-based  requirements  generation  process,  providing  a  top-down  baseline  for 
identifying  future  capabilities.  It  uses  a  Concept  of  Operations  (CONORS)  analysis 
technique  to  assess  current  systems’  and  programs’  abilities  to  provide  the  warfighter  with 
capabilities  to  accomplish  missions  envisioned  in  the  applicable  CONORS.  These  CONORS 
provide  the  basis  for  operational  scenario  development. 

Two  of  the  JCIDS  key  documents,  the  Capabilities  Design  Document  (CDD)  and 
Capabilities  Rroduction  Document  (CRD): 

state  the  operational  and  support-related  performance  attributes  of  a  system  that 
provide  the  desired  capability  required  by  the  warfighter,  attributes  so  significant  that  they 
must  be  verified  by  testing  and  evaluation.  The  documents  shall  designate  the  specific 
attributes  considered  essential  to  the  development  of  an  effective  military  capability  and 
those  attributes  that  make  significant  contribution  to  the  key  characteristics  as  defined  in  the 
[Joint  Operations  Concepts]  JOpsC  as  [Key  Rerformance  Rarameters]  KRRs.  (Chairman  of 
the  Joint  Chiefs  of  Staff,  2005,  May,  p.  A-17) 

Key  system  attributes  within  the  context  of  the  CONORS  are  the  genesis  of  scenario 
building  and  will  help  guide  the  user  in  developing  a  prioritized  set  of  operational  scenarios 
considered  essential  in  designing  the  software  architecture. 

Failure  Modes  and  Effects  Criticality  Analysis  (FMECA) 

Failure  Modes  and  Effects  Criticality  Analysis  (FMECA)  is  a  type  of  exploratory 
scenario  analysis  designed  to  expose  potential  failure  modes  and  their  impact  on  the  system 
functionality  and  mission  accomplishment.  Scenarios  are  developed  that  explore  system 
operations  in  likely  or  critical  subsystem  failure  modes;  then,  the  criticality  of  those  failures  Is 
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analyzed.  Operations  in  degraded  modes  are  also  analyzed  to  gain  insight  into  graceful 
degradation  capabilities  as  subsystems  fail  and  the  system  is  reduced  to  ever-decreasing 
levels  of  basic  functionality.  With  up  to  80%  of  weapon-system  functionality  in  the  system 
software,  it  is  critical  for  the  design  engineer  to  understand  warfighter  needs  and 
expectations  in  these  failure  modes. 

FMECA  scenarios  with  the  software  systems  and  subsystems  provide  architectural 
design  cues  to  software  engineers.  These  scenarios  provide  analysis  for  designing 
redundant  systems  for  mission-critical  elements,  “safe  mode”  operations  for  survivability- 
and  safety-related  systems,  and  drive  the  software  engineer  to  conduct  “what  if  analyses 
with  a  superior  understanding  of  failure-mode  scenarios.  For  example,  nearly  all  military 
aircraft  are  “fly-by-wire,”  with  no  physical  connection  between  the  pilot  controls  and  the 
aircraft-control  surfaces,  so  basic  software  avionic  functions  must  be  provided  in  the  event  of 
damage  or  power-loss  situations  to  give  the  pilot  the  ability  to  perform  basic  flight  and 
navigation  functions.  Obviously,  this  would  be  a  major  design  driver  for  the  software 
architect. 

Architectural  Trade-off  Analysis  s'” 

The  Software  Engineering  Institute’s  Architectural  Trade-off  Analysis  Methodology 
(ATAM)  is  an  architectural  analysis  tool  designed  to  evaluate  design  decisions  based  on  the 
quality  attribute  requirements  of  the  system  being  developed.  The  methodology  Is  a 
process  for  determining  whether  the  quality  attributes  are  achievable  by  the  architecture  as 
it  has  been  conceived  before  enormous  resources  have  been  committed  to  that  design. 

One  of  the  main  goals  is  to  gain  insight  into  how  the  quality  attributes  trade  off  against  each 
other  (Kazman,  Klein  &  Clements,  2000,  August,  p.  1). 

Within  the  Systems  Engineering  Process  (SEP),  the  ATAM  provides  the  critical 
Requirements  Loop  process,  tracing  each  requirement  or  quality  attribute  to  corresponding 
functions  reflected  in  the  software  architectural  design.  Whether  ATAM  or  another  analysis 
technique  is  used,  this  critical  SEP  process  must  be  performed  to  ensure  that  functional-  or 
object-oriented  designs  meet  all  stated,  derived,  and  implied  warfighter  requirements.  In 
complex  systems  development  such  as  weapon  systems,  half  or  more  than  half  of  the  total 
software  development  effort  will  be  expended  in  the  architectural  design  process.  Therefore, 
the  DoD  Program  Managers  must  ensure  that  the  design  is  addressing  requirements  in 
context  and  that  the  resulting  architecture  has  a  high  probability  of  producing  the 
warfighters’  capabilities  described  in  the  JCIDS  documents. 

The  ATAM  focuses  on  quality  attribute  requirements,  so  it  is  critical  to  have  precise 
characterizations  for  each.  To  characterize  a  quality  attribute,  the  following  questions  must 
be  answered: 

■  What  are  the  stimuli  to  which  the  architecture  must  respond? 

■  What  is  the  measurable  or  observable  manifestation  of  the  quality  attribute  by  which 

its  achievement  is  judged? 

■  What  are  the  key  architectural  decisions  that  impact  achieving  the  attribute 

requirement?  (2000,  p.  5) 

The  scenarios  are  a  key  to  providing  the  necessary  information  to  answer  the  first 
two  questions,  driving  the  software  engineer  to  design  the  architecture  to  answer  the  third. 
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The  ATAM  uses  three  types  of  scenarios:  Use-case  scenarios  involve  typical  uses 
of  the  system  to  help  understand  quality  attributes  in  the  operational  context;  growth 
scenarios  involve  anticipated  upgrades,  added  interfaces  supporting  system-of-systems 
development,  and  other  maturity  needs;  and  expioratory  scenarios  involve  extreme 
conditions  and  system  stressors,  including  FMECA  scenarios  (2000,  pp.  13-15).  As 
depicted  in  Figure  2,  below,  the  scenarios  build  on  the  basis  provided  in  the  JCIDS 
documents  and  requirements  developed  through  the  QAW  process.  These  processes  lend 
themselves  to  development  in  an  Integrated  Product  Team  (IPT)  environment  led  by  the 
user/combat  developer  and  including  all  of  the  system’s  stakeholders.  The  IPT  products  will 
include  a  set  of  scenarios,  prioritized  by  the  needs  of  the  warfighter  for  capability.  The 
prioritization  process  provides  a  basis  for  architecture  tradeoff  analyses.  When  fully 
developed  and  prioritized,  the  scenarios  provide  a  more  complete  understanding  of 
requirements  and  quality  attributes  in  context  with  the  operation  and  support  of  the  system 
over  its  lifecycle. 

QAW  &  ATAM  Integration  into  SW  Lifecycle  Management 
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Figure  2.  QAW  &  ATAM  Integration  into  Software  Lifecycle  Management 


Just  as  the  QAW  process  provides  a  methodology  supporting  RFP  and  Source- 
selection  activities,  the  Software  Specification  and  System  Requirements  Reviews  (SSR  and 
SRR),  the  ATAM  provides  a  methodology  supporting  design  analyses,  test  program 
activities,  the  System  Functional  and  Preliminary  Design  Reviews  (SFR  and  PDR).  The 
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QAW  and  ATAM  methodologies  are  probably  not  the  only  effective  methods  supporting 
software  development  efforts,  but  they  fit  particularly  well  with  the  DoD’s  goals,  models  and 
SEP  emphasis.  The  user/combat  developer  (blue  arrow  block  in  Figure  2,  above)  is  kept 
actively  involved  throughout  the  development  process — providing  key  insights  the  software 
developer  needs  to  successfully  develop  warfighter  capabilities  in  a  sustainable  design  for 
long-term  effectiveness  and  suitability.  The  system  development  activities  are  conducted 
with  superior  understanding  and  clarity,  reducing  scrap  and  rework,  and  saving  cost  and 
schedule.  The  technical  reviews  and  audits  (part  of  the  DoD  overarching  SEP)  are 
supported  with  methodologies  that  enhance  the  visibility  of  the  development  work  that  is 
needed  and  the  progress  toward  completing  it. 

One  of  the  main  goals  in  analyzing  the  scenarios  is  to  find  key  architectural  decision 
points  that  pose  risk  for  meeting  quality  requirements.  Sensitivity  points  are  determined, 
such  as  real-time  latency  performance  shortfalls  In  target  tracking.  Tradeoff  points  are  also 
examined,  such  as  level  of  encryption  and  message-processing  time.  The  Software 
Engineering  Institute  explains,  “Tradeoff  points  are  the  most  critical  decisions  that  one  can 
make  in  an  architecture,  which  is  why  we  focus  on  them  so  carefully”  (Kazman,  Klein  & 
Clements,  2000,  August,  p.  23). 

The  ATAM  provides  an  analysis  methodology  that  compliments  and  enhances  many 
of  the  key  DoD  acquisition  processes.  It  provides  the  requirements  loop  analysis  in  the 
SEP,  extends  the  user/stakeholder  JCIDS  involvement  through  scenario  development, 
provides  informed  architectural  tradeoff  analyses,  and  vastly  improves  the  software 
developer’s  understanding  of  the  quality  requirements  in  context.  Architectural  risk  is 
significantly  reduced,  and  the  software  architecture  presented  at  the  Preliminary  Design 
Review  (PDR)  is  likely  to  have  a  much  higher  probability  of  meeting  the  warfighters’  need  for 
capability. 

Test-case  Development 

A  significant  product  resulting  from  the  ATAM  is  the  development  of  test  cases 
correlating  to  the  use  case,  growth,  and  exploratory  scenarios  developed  and  prioritized. 
Figure  3,  below,  depicts  the  progression  from  user-stated  capability  requirements  in  the 
JCIDS  documents  to  the  ATAM  scenario  development,  and  finally  to  the  corresponding  test 
cases  developed.  The  linkage  to  the  user  requirements  is  very  strong  as  the  user 
documents  drive  the  development  of  the  three  types  of  scenarios,  and  in  turn,  the  scenarios 
drive  the  development  of  the  use  cases.  The  prioritization  of  the  scenarios  from  user-stated 
Key  Performance  Parameters  (KPPs),  Critical  Operational  Issues  (COIs),  and  FMECA 
analysis  flows  to  the  test  cases,  helping  to  create  a  system  test  program  designed  to  focus 
on  effectiveness  and  suitability  tests — culminating  in  the  system  Operational  Test  and 
Evaluation  (OT&E). 
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Figure  3.  Capabilities-based  ATAM  Scenario  Development 

The  software  developer’s  understanding  of  the  eventual  performance  required  to  be 
considered  successful  guides  the  design  of  the  architecture  and  every  step  of  the  software 
development,  coding,  and  testing  through  to  the  Full  Operational  Capability  (FOC)  delivery 
and  OT&E.  Coding  and  early  testing  of  software  units  and  configuration  items  is  much  more 
purposeful  due  to  this  level  of  understanding. 

The  resulting  test  program  is  very  comprehensive  as  each  prioritized  scenario 
requires  testing  or  other  verification  methodologies  to  demonstrate  how  the  software 
performs  in  each  related  scenario  and  satisfies  the  quality  attributes  borne  of  the  user 
requirements.  The  testing  supports  the  SEP  design  loop  by  verifying  that  the  software 
performs  the  functions  allocated  to  it  and  in  aggregate,  performs  the  verification  loop 
process  by  demonstrating  that  the  final  product  produces  the  capability  identified  in  the  user 
requirements  through  operational  testing. 

Architectural  Analysis  Products 

Architecture  Documentation  and  the  Preliminary  Design  Review 

(PDR) 

One  of  the  main  purposes  of  the  PDR  is  to  evaluate  the  system  architectural  design 
before  committing  significant  resources  to  the  construction  of  the  system.  It  is  a  key  review 
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in  the  SEP  as  it  provides  traceability  from  the  requirements  to  the  functional  allocation  of  the 
proposed  design. 

It  is  critical  to  have  a  complete  functional-  or  object-oriented  Software  Design 
Document  reviewed  at  the  PDR.  Given  that,  the  software  developer  would  likely  have  spent 
50%  or  more  of  the  effort  at  the  time  of  the  PDR  for  a  software-intensive  system. 

Discovering  that  the  proposed  software  design  is  insufficient  at  this  point  in  the  development 
cycle  can  be  disastrous  to  the  budget  and  schedule  for  the  entire  program,  especially  if  the 
proposed  design  must  be  scrapped  or  if  there  is  significant  redesign  required. 

Architecture  Documentation 

Documenting  the  process  decisions  in  designing  the  software  architecture  provides  a 
record  of  design  decisions,  tradeoffs  made,  and  priorities  implemented  throughout  the 
design  effort  and  design  reviews.  The  active  involvement  of  the  user  and  all  system 
stakeholders  throughout  this  process  is  one  of  the  keys  to  achieving  a  robust  design  that 
provides  warfighter  capabilities  and  long-term,  cost-effective  sustainability.  The  ATAM 
provides  methodologies  that  formalize  the  stakeholder  participation  in  the  architectural 
design. 


The  ATAM  would  help  drive  documentation  from  quality  attributes  to  both  the  three 
types  of  prioritized  scenarios  as  weii  as  the  test  cases  needed  to  demonstrate  or  verify 
performance.  The  quality  attributes  are  understood  in  the  context  of  the  user-prioritized 
scenarios,  so  design  decisions  have  strong  linkage  to  user  priorities.  The  test  cases  help 
guide  the  design  effort  as  the  software  engineer  has  a  very  clear  understanding  of  what  the 
software  must  do,  under  what  conditions,  and  to  what  standard.  Design  reviews  each  have 
a  clearly  defined  focus,  with  the  ATAM  products  providing  a  common  understanding  of  what 
is  to  be  accomplished. 

Scenario  Inventory 

One  of  the  main  products  resulting  from  the  ATAM  is  the  prioritized  inventory  of  use 
case,  growth,  and  expioratory  scenarios  that  drive  the  architecturai  design.  As  the  user 
(along  with  other  stakeholders)  is  the  primary  source  for  scenario  development,  the  resulting 
design  is  user-oriented,  not  engineer-oriented. 

The  prioritization  of  the  scenarios  provides  the  basis  for  tradeoff  analyses  and  design 
decisions,  placing  tradeoff  decisions  where  they  should  be — with  the  warfighter.  With  the 
user  involved  throughout  the  design  process,  the  resuiting  system  is  much  more  iikeiy  to 
satisfy  warfighter  capability  requirements. 

Software  and  System  Test  Program 

The  development  of  test  cases  from  the  scenarios,  as  depicted  in  Figure  3  above, 
provides  the  Design  Loop  function  of  the  SEP  by  ensuring  that  the  software  deveioped 
performs  the  functions  defined  by  the  scenarios,  which  represent  the  quality  attribute 
requirements  in  context.  The  inventories  of  test  cases  are  developed  from  the  user-defined 
scenarios  so  that  there  is  one  or  more  test  case  for  every  scenario.  The  test  cases  wiil  tend 
to  satisfy  both  technicai  issues  (as  the  software  developed  will  be  tested  against  its  intended 
function)  as  weii  as  operational  issues,  as  each  function  is  borne  of  the  users’  scenarios. 
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The  aggregated  test  cases  are  part  of  the  system’s  overall  test  program  and 
contribute  to  readiness  for  the  Initial  Operational  Test  and  Evaluation  (lOT&E).  The  lOT&E 
is  the  defining  event  in  the  SEP  Verification  Loop,  ensuring  that  the  software  developed 
satisfies  user  effectiveness  and  suitability  requirements  and  meets  warfighter  capability 
needs  specified  in  the  JCIDS  documents. 

Software  Design  Metrics 

From  the  DoD’s  point  of  view,  gaining  insight  and  control  of  the  software  design 
process  is  crucial  to  delivering  the  warfighter  capabilities  required.  In  addition,  metrics 
provide  a  means  to  monitor  and  control  the  process.  The  metrics  chosen  must  provide  the 
DoD  insight  into  how  the  software  architecture  is  designed  to  satisfy  quality  attributes  and 
requirements  across  a  broad  spectrum  of  functionality  and  long-term  sustainability 
performance.  In  addition,  technically  oriented  design  metrics  such  as  complexity  are  also 
important,  but  are  not  the  focus  of  this  research. 

The  system  architectural  design  is  very  much  a  shared  responsibility  between  the 
DoD  and  the  software  developer,  so  metrics  must  also  reflect  developmental  measures 
spanning  both.  For  instance,  designating  the  completed  set  of  prioritized  scenarios  as  a 
design  metric  involves  measuring  the  build  of  the  scenarios  in  a  collaborative 
user/stakeholder/developer  environment. 

Using  the  completion  of  the  ATAM  products  as  metrics  is  logical  as  they  are 
measurable,  are  key  processes  in  the  architectural  design,  and  serve  as  indicators  to  the 
progress  towards  successfully  completing  the  design  process.  Useful  ATAM-based  metrics 
would  include: 

■  Business  Drivers  Developed 

■  Prioritized  Scenario  Sets  Developed 

■  Attribute  Utility  Tree 

■  Sensitivity  Points  &  Tradeoff  Points 

■  Architecture  Approach  Document 

Summary 

The  main  goal  of  the  DoD  acquisition  process  is  to  develop  identified  warfighter 
capabilities  within  predicted  and  controlled  timelines  and  cost  targets;  yet,  many  software¬ 
intensive  systems  developed  have  experienced  significant  cost  and  schedule  growth  due,  at 
least  in  part,  to  the  software  development  component.  There  are  many  factors  that 
contribute  to  the  problem — including  how  and  when  the  DoD  conveys  the  needed  quality 
attribute  requirements. 

The  DoD  acquisition  model  uses  the  Systems  Engineering  Process  (SEP)  as  the 
central  process  for  controlling  the  developmental  process  of  its  systems.  The  SEP  is  an 
integrated  process  with  the  DoD  and  the  contractors  selected,  thereby  urging  shared 
responsibility  for  effective  systems  development.  The  process  begins  and  ends  with  the 
user  or  combat  developer  responsible  for  providing  the  capabilities-based  requirements, 
which  are  further  developed  and  decomposed  by  the  Program  Manager  and  contractors 
responsible  for  building  the  system.  The  system  components  are  constructed,  integrated 
and  continually  tested,  culminating  in  the  User’s  acceptance  testing,  usually  the  Initial 
Operational  Test  and  Evaluation  (lOT&E). 
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A  key  to  the  SEP  implementation  is  effective  and  compiete  deveiopment  and 
communication  of  the  system  requirements.  This  must  happen  at  some  point  for  any  system 
to  be  successfuily  developed;  but  when  it  happens  is  extremely  important  to  the  cost  and 
schedule  estimate  accuracy.  When  the  contractor  has  a  good  understanding  of  the  work  to 
be  completed  from  the  requirements  presented,  more  accurate  estimates  are  offered  in  the 
contractor’s  proposai  before  the  program  scheduie  is  locked  in  with  a  contract.  If  a 
significant  portion  of  the  work  is  discovered  through  requirements  decomposition  after  the 
contract  is  in  place  (typical  of  software  components),  the  estimates  provided  in  the  proposal 
are  severely  understated,  and  the  program  schedule  and  budgets  are  no  longer  appropriate. 

One  reason  the  software  component  is  more  sensitive  to  the  requirements 
development  is  that  the  software  engineering  environment  is  immature  when  compared  to 
most  hardware-centric  environments.  Vague  or  missing  requirements  for  a  hardware  item 
may  be  compensated  by  a  mature  engineering  environment  that  accommodates  implied 
requirements.  For  instance,  the  automotive  industry  would  provide  the  ability  to  easily 
replace  normal  wear-out  items  like  filters  and  tires,  whether  or  not  such  provisions  were 
specified.  The  software  engineering  environment  does  not  offer  that  level  of  maturity. 

The  MUIRS  analytical  technique  helps  capture  software  performance  requirements 
that  are  routinely  overlooked  in  the  immature  software  engineering  environment.  The 
MUIRS  analysis  helps  capture  and  convey  Open  Architecture  needs,  safety  and  security 
considerations,  and  long-term  supportability  performance  needed  by  the  warfighter. 

In  addition  to  simply  understanding  the  breadth  of  system  requirements,  the  software 
engineer  needs  to  understand  them  in  context  of  the  operations,  supportability,  and 
environments  to  design  a  software  architecture  that  is  effective.  It  is  not  enough  to 
understand  what  the  software  must  do;  the  engineer  must  understand  under  what 
circumstances,  in  what  environments,  and  to  what  standard  the  function  must  be  performed. 

What  the  DoD  needs  to  improve  the  acquisition  of  software-intensive  systems  are 
methodologies  that  capture  and  convey  quality  attribute  requirements  in  an  operational 
context,  within  a  Systems  Engineering  Process  environment.  The  Software  Engineering 
Institute’s  Quality  Attribute  Workshop  (QAW)  and  Architecture  Tradeoff  Analysis 
Methodology  (ATAM)  provide  well-suited  techniques  for  developing  requirements  in 
context.  The  QAW  process  before  contracting  helps  provide  enough  requirements 
elicitation  for  more  accurate  contractor  proposals;  likewise,  the  ATAM  helps  provide  the 
operational  context  through  scenario  and  test-case  development  before  the  software  design 
effort.  Both  products  support  the  SEP,  providing  methodologies  for  performing  critical  SEP 
functions. 

DoD  personnel  (user/combat  developer  and  Program  Manager/materiel  developer) 
are  key  and  integral  to  the  development  of  effective  and  suitable  warfighter  capabilities 
within  predictable  cost  and  schedule  parameters.  Improving  the  processes  that  develop  and 
convey  system  quality  attribute  requirements  in  context  will  improve  the  cost,  schedule  and 
performance  predictability  of  software-intensive  systems  and  will  reduce  the  supportability 
costs  over  the  life  of  the  system. 
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Abstract 

The  Navy  is  currently  implementing  the  open-architecture  framework  for  developing 
joint  interoperable  systems  that  adapt  and  exploit  open-system  design  principles  and 
architectures.  This  raises  concerns  about  how  to  practically  achieve  dependability  in 
software-intensive  systems  with  many  possible  configurations  when:  1)  the  actual 
configuration  of  the  system  is  subject  to  frequent  and  possibly  rapid  change,  and  2)  the 
environment  of  typical  reusable  subsystems  is  variable  and  unpredictable.  Our  preliminary 
investigations  indicate  that  current  methods  for  achieving  dependability  in  open 
architectures  are  insufficient.  Conventional  methods  for  testing  are  suited  for  stovepipe 
systems  and  depend  strongly  on  the  assumptions  that  the  environment  of  a  typical  system  is 
fixed  and  known  in  detail  to  the  quality-assurance  team  at  test  and  evaluation  time.  This 
paper  outlines  new  approaches  to  quality  assurance  and  testing  that  are  better  suited  for 
providing  affordable  reliability  in  open  architectures,  and  explains  some  of  the  additional 
technical  features  that  an  Open  Architecture  must  have  in  order  to  become  a  Dependable 
Open  Architecture. 

Introduction 

The  Navy’s  Open  Architecture  (OA)  is  defined  to  be  a  multi-faceted  strategy 
providing  a  framework  for  developing  joint  interoperable  systems  that  adapt  and  exploit 
open-system  design  principles  and  architectures  (DAU,  2007b).  The  objective  of  supporting 
adaptable  systems  has  significant  implications  for  quality  assurance.  OA  approaches  often 
involve:  (i)  a  public,  non-proprietary  architecture  that  can  accept  plug-in  components  and  be 
transparent  to  changes  (e.g.,  the  system  should  continue  to  work  if  selected  components  or 
connectors  are  replaced  by  different  components  or  connectors),  and  (ii)  an  architecture 
whose  purpose  is  to  make  explicit  the  common  interfaces  (e.g.,  POSIX,  CORBA,  etc.).  Main 
goals  of  Navy’s  OA  include  minimizing  total  cost  of  ownership,  increasing  competition, 
achieving  reuse,  optimizing  systems,  and  developing  systems  that  support  evolution. 

This  paper  explores  some  test  and  evaluation  implications,  outlines  an  approach  for 
providing  affordable  quality  assurance  in  the  kind  of  dynamic  environment  that  open 
architectures  are  intended  to  accommodate,  and  evaluates  the  current  state  of  some 
technologies  that  support  the  new  approach. 

The  Navy's  requirements  to  implement  OA  are  set  forth  in  several  Department  of 
Defense  (DoD)  and  Department  of  Navy  (DoN,  2004,  August  5)  policy  documents  (e.g.,  “The 
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Defense  Acquisition  System”  (DoD,  2003,  May  12),  “Guidance  Regarding  Modular  Open 
Systems  Approach  (MOSA)  Implementation”  (DoD,  2004,  April  5),  “Naval  Open  Architecture 
Scope  and  Responsibilities”  (DoN,  2004,  August  5),  etc.).  In  the  past  the  Navy  has  acquired 
systems  that,  although  they  performed  their  functions  and  tasks  exceedingly  well,  were 
unique  in  their  designs  and  engineering.  Indeed,  they  required  unique  parts,  equipment,  and 
services  to  support  them,  were  supported  by  a  limited  number  of  suppliers  and  became 
unaffordable  to  maintain.  There  are  numerous  instances,  moreover,  in  which  a  system  or 
platform  was  scrapped  rather  than  upgraded  or  modernized  because  the  cost  to  do  so  had 
become  prohibitive.  Test  and  evaluation  account  for  an  appreciable  part  of  the  cost  for 
system  upgrades.  This  paper  explores  how  open  architecture  principles  can  be  extended 
and  applied  to  reduce  these  costs  and  to  make  Navy  systems  more  agile. 

Business  issues  are  pushing  the  Navy  to  shift  its  development  processes  towards  an 
open-architecture  paradigm.  In  an  era  of  strenuous  competition  for  dollars,  the  Navy  is 
continuously  challenged  with  budget  decisions.  Inflexible  acquisition  strategies  lock  the 
Navy  Into  single  systems  and  vendors  that  limit  the  service's  options  for  competition  and 
innovation.  Limited  competition  impedes  innovation,  while  OA  provides  options  for  greater 
competition  and  inclusion  of  innovators.  Cost  of  procured  systems  is  due  to  maintenance  as 
well  as  development  expense.  Stovepiped  processes  lead  to  acquisition  of  systems  across 
the  Navy  with  duplicated  capabilities.  For  example,  every  ship  (class)  had  a  unique  combat 
system.  Currently,  limited  asset  reuse  takes  place  across  the  enterprise  without  open 
architectures.  However,  there  are  few  enterprise  processes  to  foster  integration  in  a  legacy 
environment.  To  achieve  rapid  fielding  of  new  technology  and  capability  for  the  Fleet,  the 
Navy’s  business  model  has  to  change  from  the  classic  acquisition  system  to  a  process  that 
supports  Rapid  Capability  Insertion.  Open  architecture  meets  those  needs  by  shortening 
cycle-times  for  getting  capability  to  the  warfighter  when  needed.  The  use  of  modular 
systems  to  facilitate  technology  refreshment  and  obsolescence  mitigation  is  a  key  aspect  of 
OA.  Increased  competition  and  innovation  are  possible  through  changed  business  practices 
enabled  by  OA. 

Many  technical  issues  are  also  motivating  the  Navy’s  change  towards  open 
architectures: 

■  Procurement  of  monolithic  systems  using  legacy  processes  produces  incompatible 

systems  that  are  not  interoperable. 

■  Software  closely  coupled  (integral)  to  the  computing  hardware  platforms  is  not 


reusable. 


■  Special-use  code  and  modules  that  cannot  be  reused  across  the  Navy  are  artifacts 
of  the  legacy  approach  to  systems  acquisition. 

■  Proliferation  (and  resulting  lifecycle  cost  growth)  of  hardware  and  software  baselines 
results  from  upgrade  processes  in  closed  systems. 

Consequently,  there  has  been  much  attention  to  cultural  issues  and  acquisition 
policies  to  facilitate  adoption  of  an  open  architecture  paradigm  for  Navy  systems. 

This  paper  addresses  a  complementary  effort  to  identify  current  weaknesses  and 
gaps  in  the  state  of  the  knowledge  with  respect  to  assuring  reliability  of  DoD/DoN  systems 
developed  according  to  open-systems  principles,  and  to  develop  or  adapt  new  methods  for 
overcoming  those  weaknesses  so  they  can  be  used  in  Navy  open  architectures.  We  are 
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studying  weaknesses  in  current  best  practices  with  respect  to  the  context  identified  above, 
and  are  performing  research  to  extend  and  develop  methods  to  overcome  those 
weaknesses. 

Our  preliminary  investigations  indicate  that  current  methods  for  achieving 
dependability  in  open  architectures  are  insufficient.  The  main  problem  is  how  to  practically 
achieve  dependability  in  software-intensive  systems  with  many  possible  configurations  when 
the  actual  configuration  of  the  system  is  subject  to  frequent  and  possibly  rapid  change,  and 
the  environment  of  typical  reusable  subsystems  is  variable  (used  in  many  platforms)  and 
unpredictable  (mission-dependent).  This  is  a  major  problem  for  practical  development 
because  real  development  projects  depend  heavily  on  software  testing,  which  is  strongly 
context-dependent.  Conventional  methods  for  testing  depend  strongly  on  the  assumptions 
that  the  environment  of  a  typical  system  Is  fixed  and  known  in  detail  to  the  quality-assurance 
team  at  test  and  evaluation  time.  These  assumptions  are  quite  reasonable  for  stovepipe 
systems  but  are  not  valid  for  open  architectures.  A  component  in  an  open  architecture 
should  be  reusable  not  only  across  current  classes  of  ships  but  also  across  future  platforms 
that  are  yet  to  be  designed — those  that  belong  to  different  services,  and  perhaps  even  to 
coalition  partners.  This  set  of  contexts  is  very  large  in  practice,  is  open-ended,  and  cannot 
even  in  principle  be  known  in  detail  to  the  test  and  evaluation  team. 

This  paper  outlines  new  approaches  to  quality  assurance  and  testing  that  are  better 
suited  for  providing  affordable  reliability  in  open  architectures,  and  explains  some  of  the 
additional  technical  features  that  an  Open  Architecture  must  have  in  order  to  become  a 
Dependable  Open  Architecture,  i.e.,  one  that  can  support  reuse  and  rapid  reconfiguration 
via  module  swapping  (without  compromising  reliability)  while  remaining  economically  viable 
at  the  level  of  individual  systems  and  reducing  total  ownership  cost  for  the  enterprise.  This 
requires  linking  the  architecture  with:  1)  specific  dependability  requirements,  2)  certifiable 
technical  standards  for  each  interaction  path,  3)  specialized  types  of  testing,  as  well  as 
combining  that  testing  with  other  kinds  of  computer-aided  quality-assurance  methods.  The 
paper  explains  the  concepts  behind  the  approach  and  why  it  is  expected  to  work  as  claimed. 

Navy’s  Vision  of  Open  Architecture 

The  Navy  Open  Architecture  (Navy  OA)  is  a  Navy  initiative  for  a  multi-faceted 
strategy  providing  a  framework  for  developing  joint  interoperable  systems  that  adapt  and 
exploit  open-system  design  principles  and  architectures  (DAU,  2007a,  DAD,  2007b).  This  is 
a  systems  design  approach  consistent  with  several  governmental  concepts  and  initiatives, 
such  as  the  Open  Architecture  Computing  Environment  (OACE)  (Naval  Sea  Systems 
Command,  2007),  FORCEnet  (FORCEnet,  2007a),  and  the  Modular  Open  Systems 
Approach  (MOSA)  (Open  Systems  Joint  Task  Force,  2007).  OACE  seeks  to  ease  the  test 
and  evaluation  burden  by  limiting  hardware  choices  to  certain  approved  possibilities. 
FORCEnet  is  an  operational  concept  that  can  benefit  from  realization  of  OA  goals  for  its 
implementation.  MOSA  is  a  joint-acquisition  approach  that  shares  many  of  the  goals  of  the 
Navy’s  OA  effort. 

The  OACE  (NSWCDD,  2004,  August  23a,  NSWCDD,  2004,  August  23b)  aims  to 
implement  open  specifications  for  interfaces,  services  and  supporting  formats.  It  enables 
software  components  to  work  across  a  range  of  systems  and  interoperate  with  other 
software  components  on  local  and  remote  systems.  Thus,  the  OACE  framework  includes  a 
set  of  principles,  processes,  and  best  practices.  The  OACE  is  a  surface-Navy  approach  to 
setting  technical  standards  for  shipboard  systems.  It  shares  many  of  the  objectives  of  OA, 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  -  287  - 


but  does  not  address  business  processes  that  deal  with  those  objectives,  and  does  not 
apply  to  submarines,  aircraft  or  C4I  systems.  The  OACE  consists  of  a  set  of  documentation 
describing  an  infrastructure  of  technologies  supported  by  a  reference  architecture.  This 
infrastructure  includes  cable  plant,  cabinets,  network  components,  processors,  operating 
systems,  adaptation  and  distribution  middleware,  frameworks,  resource  management, 
common  services  (e.g.,  system  server  applications  such  as  web  servers),  etc.  As  an 
example.  Figure  1  shows  the  reference  OA  defined  by  the  OACE. 


Figure  1.  The  OACE’s  Open  Architecture  Computing  Environment 

(extracted  from  NSWCDD,  2004,  August  23a) 


The  OACE  also  defines  guidance  and  strategies  for  fault  tolerance,  scalability, 
portability,  real-time  performance,  system  composition,  system  test  &  certification,  and 
selection  of  standards  (e.g.,  POSIX,  CORBA,  etc.).  The  OACE  will  allow  the  Navy  to 
introduce  and  change  out  commercial  technology  to  maximize  affordability  and  performance 
goals. 


FORCEnet  is  the  operational  construct  and  architectural  framework  for  Naval 
Warfare  in  the  Information  Age  to  integrate  warriors,  sensors,  networks,  command  and 
control,  systems,  platforms,  and  weapons  into  a  networked,  distributed  combat  force, 
scalable  across  the  spectrum  of  conflict  from  seabed  to  space  and  sea  to  land  (FORCEnet, 
2007a).  FORCEnet  Is,  thus,  the  future  implementation  of  the  Network  Centric  Warfare  in  the 
Navy,  and  is  the  Navy's  primary  effort  to  integrate  multiple  architecture  and  standards 
efforts.  Research  efforts  demonstrated  that  across  the  Navy  Enterprise,  FORCEnet  viability, 
affordability  and  sustenance  necessitates  an  architecture  that  is  in  full  compliance  with  OA 
technology,  systems  and  standards.  The  development  and  embedding  of  OA  within 
FORCEnet  will  enable  a  superior,  adaptive,  “plug  and  fight”  capability  for  the  modern 
warfighter  of  today  and  tomorrow.  Figure  2  presents  the  system  interface  view  of 
FORCEnet. 
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Figure  2.  FORCEnet’s  System  Interface  Description 

(extracted  from  FORCEnet,  2007b) 


The  Naval  Open  Architecture  Enterprise  Team  (OAET)  is  currently  spearheading  an 
OA/FORCEnet  Risk  Reduction  Experimentation  effort  to  minimize  the  risk  of  delivering 
interoperable  products  (Shannon,  2006).  This  effort  is  in  its  early  stages  and  has  recently 
completed  its  first  cycle.  An  example  of  a  project  enabling  the  integration  of  OA  into 
FORCEnet  is  the  “Open  Architecture  as  an  Enabler  for  FORCEnet”  project  (Deering  et  al., 
2006,  September).  It  concentrates  on  implementing  network-centric  military  operations  with 
specific  threat-engagement  scenarios  (i.e.,  sensed  threats  to  available  weapons).  These 
concepts  are  applied  to  the  FORCEnet  OA  Domain  Model  using  legacy  and  future 
warfare/Navy  systems  based  on  OA  concepts.  An  analysis  exposed  potential  functional 
boundary  limitations  in  the  current  OA  Domain  model,  and  a  revised  model  has  been 
proposed. 

The  Modular  Open  Systems  Approach  (MOSA)  (Open  Systems  Joint  Task  Force, 
2007)  is  both  a  business  and  technical  strategy  for  developing  new  systems  or  modernizing 
existing  ones  (see  Figure  3). 
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Figure  3.  MOSA’s  Fundamental  Building  Blocks 

(extracted  from  Flowers  &  Azani,  2004) 


As  a  business  strategy,  the  MOSA  enables  program  teams  to  build,  upgrade  and 
support  systems  more  quickly  and  affordably.  This  can  be  achieved  through  the  use  of 
commercial  products  from  multiple  sources  and  by  leveraging  the  commercial-sector 
investment  in  new  technology  and  products.  The  technical  portion  of  the  MOSA  addresses  a 
system  design  that  is  modular,  has  well-defined  interfaces,  is  designed  for  change  and,  to 
the  extent  possible,  makes  use  of  commonly  used  industry  standards  for  key  interfaces.  This 
system  design  is  best  accomplished  using  collaborative  engineering  based  on  sound 
systems  engineering  processes.  Adherence  to  MOSA  allows  for  developing  DoD  systems 
that  account  for  the  growing  asymmetrical  threats,  unprecedented  rate  of  technological 
change,  and  requirements  for  joint  warfighting  capabilities.  The  Navy’s  OA  is  closely  related 
to  MOSA.  OA  is  a  more  specific  extension  of  generalized  MOSA  principles.  Naval  OA 
applies  to  computer-intensive  National  Security  Systems  as  defined  in  the  Clinger  Cohen 
Act,  while  MOSA  has  broader  applicability,  e.g.,  including  mechanical  systems. 

The  successful  implementation  of  OA  principles  in  the  Navy  may  bring  multiple 
benefits  from  both  business  and  technical  viewpoints  to  the  Navy  and  other  DoN/DoD 
organizations.  Business  benefits  include:  (i)  enterprise-wide  plans  based  on  a  cost/capability 
analysis  of  programs  that  address  capability,  affordability,  and  stabilization,  (ii)  flexible 
acquisition  strategies  and  contracts  that  enable  the  Navy  to  reuse  software,  easily  upgrade 
systems,  and  share  data  throughout  the  enterprise,  (iii)  streamlined  investments  in  similar 
capabilities,  (iv)  increased  competition  to  foster  innovation  and  leverage  technology 
upgrades,  and  (v)  established  enterprise  processes  and  governance  to  foster  integration. 

On  the  other  hand,  an  efficient  implementation  of  OA  principles  yields  many  technical 
benefits,  including:  (i)  layered  and  modular  open  architectures  that  address  portability, 
maintainability,  interoperability,  upgradeability  and  long-term  supportability,  (ii)  modular, 
open  designs  consisting  of  components  that  are  self-contained  elements  with  well-defined 
interfaces,  (iii)  maximum  use  of  commercial  standards  and  commodity  “commercial  off-the- 
shelf  (COTS)  products,  and  (iv)  systems  that  continuously  conform  with  Information 
Assurance  (lA)  requirements  and  monitor  technology  developments  for  lA  improvements. 
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Figure  4  below  presents  a  synthesis  of  OA  benefits. 


Performance 

•  Continuous  competition  yeilds  best  of  breed  applications 

•  Focus  on  warfighting  priorities 

Schedule 

•  System  integration  of  OA  compliant  software  happens  quickly 

•  Rapid  update  deliveries  driven  by  use  operational  cycles 

Cost  Avoidance 
Mechanisms 

•  Software  -  develop  once,  use  often,  upgrade  as  required 

•  Hardware  -  Use  high-volume  COTS  products  at  optimum  price 
points 

•  Training  systems  use  same  tactical  applications  and  COTS 
hardware 

Design  for  Maintenance 
Free  Operating  Periods 

•  Install  adequate  processing  power  to  support  "fail-over"' 
without  maintenance 

•  Schedule  replacement  with  improved  COTS  vice  maintaining 
old  hardware 

•  Reduce  maintenance  training  required 

•  Consolidate  Development  and  Operatioinal  Testing  for  reused 
applications 

Risk  Reduction 

•  Field  new  applications  only  when  mature 

•  Do  not  force  the  last  ounce  of  performance 

•  Deploy  less  (but  still  better  than  existing)  performance  or  wait 
until  next  update 

Figure  4.  Open  Architecture  Benefits 

(extracted  from  DAU,  2007C) 

The  Navy  still  needs  to  complete  carrying  out  the  necessary  business  and  technical 
changes  to  achieve  the  stated  OA  goals.  Well-known  technical  changes  include  the  need  for 
continuing  the  transition  to  COTS-based  computing  plants  in  modular  architectures,  the 
development  of  an  OA  Enterprise  component  library  capability  to  facilitate  market  research 
and  reuse  of  components,  the  alignment  of  standards  among  the  domains  and  the 
alignment  of  standards  to  the  DoD  Information  Technology  Standards  Registry  (DISR).  This 
paper  identifies  additional  technical  changes  related  to  test  and  evaluation. 

Difficulties  in  Testing  Systems  with  Open  Architectures 

The  Navy  has  emphasized  improving  its  business  and  organizational  processes, 
structure  and  expertise  over  technical  matters.  The  Navy  is  currently  able  to  deliver  open 
architecture-based  systems.  However,  known  methods  for  achieving  dependability  with  OA 
are  expensive  and  not  clearly  understood.  The  Navy’s  current  approach  to  system  testing  is 
not  well  matched  to  the  needs  of  open  environments.  It  is  too  expensive;  it  takes  too  long, 
and  it  lacks  agility  to  react  to  changes  during  and  after  acquisition. 

Traditional  testing  techniques,  such  as  scenario-based  testing,  are  commonly  used 
for  assessing  dependability  of  Navy  systems.  These  techniques  are  strongly  dependent  on  a 
particular  system  configuration  and  environment.  The  environment  is  usually  modeled  using 
flat,  uniform  distributions  of  software  inputs  and  a  limited  number  of  profiles.  Accordingly,  the 
environment’s  profile  and  the  most  relevant  estimates  of  the  application  inputs  are 
considered.  For  example,  in  Navy’s  control  systems,  input  parameters  such  as  the  number 
of  weapons  or  the  number  of  strike  elements  are  Included  within  the  testing  profiles. 

The  drawback  of  these  techniques  is  that  when  the  system  configuration  or 
environment  changes,  the  designed  test  cases  also  need  to  be  changed.  Plugging  in  a  new 
component  will  lead  to  a  completely  different  system  and  will  likely  invalidate  the  test 
scenarios  and  profiles  previously  used.  A  similar  problem  also  occurs  when  the  application 
has  to  be  used  in  an  operational  environment  other  than  the  one  for  which  it  was  originally 
designed,  which  is  expected  to  be  common  for  reusable  components.  This  raises  an 
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important  concern  since  Navy  systems  are  submitted  to  frequent  changes.  Better  ways  of 
doing  testing  and  evaluation  are,  thus,  highly  desirable. 

Acquisition  of  new  system  modules  and  components  is  also  an  important  concern. 
While  an  architectural  or  modular  approach  should  allow  for  a  certain  degree  of 
predictability,  current  Navy  testing  processes  do  not  deal  with  modularity.  As  a  result,  time- 
consuming  and  expensive  test  procedures  are  needed  each  time  a  new  system  release 
comes  up  because  available  testing  methods  cannot  support  the  high  frequency  of  releases. 
Methods  that  limit  the  number  of  configurations  of  an  architecture  might  be  required,  at  least 
in  the  near  term.  Such  limitations  may  be  able  to  be  relaxed  as  technologies  for  testing 
families  of  systems  improve. 

In  flexible,  open  systems,  components  need  to  be  assembled  in  a  large  number  of 
configurations;  and  because  the  system  is  open,  new  components  can  be  added  that  did  not 
even  exist  at  the  time  the  system  was  originally  designed. 

In  practice,  the  number  of  possible  configurations  for  an  open  system  is  very  large, 
because  each  of  many  slots  in  an  open  architecture  can  be  independently  filled  by  several 
different  specific  subsystems.  Because  the  number  of  choices  for  each  slot  must  be 
multiplied  together  to  produce  the  total  number  of  possible  configurations,  the  number  of 
possibilities  is  astronomical  for  the  kinds  of  systems  designed  by  the  Navy.  For  example,  it 
has  been  estimated  that  avionics  software  systems  have  thousands  of  components  and  tens 
of  thousands  of  connections.  In  principle,  the  number  of  configurations  is  unbounded 
because  an  unknown  and  unlimited  number  of  new  subsystems  can  be  created  in  the  future. 
One  consequence  of  this  is  that  it  will  be  impossible  to  test  all  configurations,  and  that  a 
majority  of  the  possible  configurations  will  not  be  tested  at  all.  These  ideas  are  graphically 
summarized  in  Figure  5. 


Figure  5.  Example  of  Various  Sets  of  Configuration  Types:  Dependable  vs.  Non- 
dependable,  Known  vs.  Unknown,  Tested  vs.  Untested 

Each  node  in  the  figure  represents  a  possible  configuration  of  an  open  architecture. 
The  connections  between  the  nodes  represent  transitions  between  possible  configurations, 
such  as  those  resulting  from  the  replacement  of  a  subsystem  with  another  plug-compatible 
subsystem  that  fits  in  the  same  slot  of  the  open  architecture.  The  figure  is  valid  at  many 
different  scales;  a  module  can  be  as  small  as  a  single  data  item,  software  procedure,  or 
integrated  circuit  chip,  and  can  include  subsystems  as  large  as  entire  ships.  Figure  5  also 
highlights  two  important  concerns  (indicated  by  the  interrogation  point):  the  non-dependable 
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configurations  that  are  unknown  and  the  non-dependable  configurations  that  (although 
known)  have  not  been  tested.  Indeed,  the  number  of  configurations  in  practice  is  too  large  to 
be  able  to  either  know  or  test  all  of  them.  We  seek  alternative  methods  to  palliate  these 
issues. 

These  considerations  indicate  that  quality  assurance  methods  that  depend  on 
checking  the  individual  possible  behaviors  of  the  entire  system  do  not  have  any  hope  of 
being  effective  for  the  test  and  evaluation  of  flexible  open  architectures,  and  that 
conceptually  new  and  different  methods  will  have  to  be  employed  to  achieve  dependability 
for  such  systems  in  the  presence  of  reuse  and  reconfiguration. 

Testing  of  reusable  subsystems  is  also  subject  to  the  above  considerations  and, 
similarly,  requires  new  methods  for  effectively  achieving  dependability.  This  conclusion  is 
consistent  with  past  experience  with  system  failures  in  military,  scientific  and  commercial 
applications.  The  majority  of  observed  failures  are  due  to  requirements  and  specifications 
errors,  many  of  which  manifest  after  a  subsystem  has  been  moved  to  a  different 
environment  than  the  one  for  which  it  was  originally  designed  and  tested.  This  is  an 
indication  that  in  current  practice,  the  effectiveness  of  testing  is  very  sensitive  to  the 
expected  operating  environment,  which  is  unknown  for  reusable  subsystems.  Indeed, 
software  reuse  may  invalidate  the  operational  profiles  and  test  cases  and  scenarios 
originally  developed.  The  new  operational  profiles,  test  cases  and  scenarios  are  unknown, 
and  no  efficient  method  exists  to  calculate  the  required  “delta”  describing  the  necessary 
changes  from  previously  used  profiles  (or  test  cases  or  scenarios)  so  that  they  can  be 
applied  to  the  newly  reconfigured  system.  Open  Architecture  facilitates  software  reuse, 
which  adds  weight  to  this  issue. 

Test  cases  correspond  to  the  traditional  artifact  used  in  testing,  which  are  based  on  a 
model  of  the  system  environment.  In  stovepipe  systems,  requirements  analysis  and  testing 
is  greatly  simplified  compared  to  open  systems.  Also,  there  exist  numerous  methods  and 
techniques  that  allow  for  linking  the  testing  results  to  dependability  parameters,  so  as  to 
obtain  a  quantitative  measure  of  the  overall  dependability  of  the  system  (e.g.,  notion  of 
“dependability  benchmarking”). 

The  traditional  concept  of  system  design  is  not  focused  on  architectural  “bits.”  An 
architecture  is  related  to  a  family  of  systems,  while  a  design  is  traditionally  associated  with  a 
single  instance  of  a  system.  Also,  an  architecture  involves  more  complexity  than  the 
traditional  notion  of  system  “configuration.”  This  is  due  to  the  fact  that  the  “context”  is 
included  in  the  architecture,  which  is  usually  unknown,  not  well  understood  or  difficult  to 
accurately  take  into  account. 

The  type  of  dependability  properties  to  be  tested  is  also  an  important  concern. 

Making  Navy  systems  dependable  will  require  considering  a  certain  level  of  system 
performance  and  availability  as  part  of  the  dependability  concern.  Indeed,  architectural 
changes  can  considerably  impact  Key  Performance  Parameters  (KPP),  availability  and  other 
system  requirements.  Other  concerns  relate  to  how  testing  can  be  applied  to  Navy  systems 
that  are  based  on  migrating  services  (e.g.,  reconfiguration  of  service-based  architectures) 
and  how  system  developers  and  testers  can  be  involved  in  the  acquisition  process.  At  the 
moment,  it  is  not  possible  to  accurately  know  how  much  it  may  cost  to  move  towards  an 
open  architecture  paradigm. 
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Proposed  Approach 


In  the  short  term,  the  problems  outlined  in  the  previous  section  are  being  addressed 
by  attempting  to  predict  future  needs  and  by  iimiting  the  allowed  configurations  accordingiy. 
This  has  the  advantage  of  minimizing  impact  on  current  development  processes  and 
organizations,  and  the  disadvantages  that  cost  of  testing  is  siil  large  and  proportional  to  the 
number  of  reconfigurations,  and  that  in  cases  in  which  predictions  of  future  needs  turn  out  to 
be  incorrect,  reconfigurations  will  need  time  for  lengthy  retesting,  or  new  configurations  will 
have  to  be  fielded  without  assurance  of  dependability.  However,  in  the  Navy’s  Open 
Architecture  vision,  the  “plug  and  fight”  process  is  supposed  to  be  inexpensive  and  agile. 

The  main  objective  of  the  OA  approach  is  to  get  away  from  monolithic  designs  and 
architectures,  and  gain  the  ability  to  replace  bits  of  systems.  The  goal  is  to  facilitate 
DoD/Navy  systems  acquisition.  This  requires  a  shift  from  scenario-based  testing  to 
architecture-based  testing.  The  constraints  expressing  the  most  important  dependability 
properties  should  be  part  of  the  architecture.  The  architecture  should,  thus,  include  not  only 
components  and  connections  but  also  constraints.  Note  that  there  are  different  types  of 
constraints — encompassing  requirements,  capabilities  and  standards  (capabilities  are 
similar  to  requirements).  A  dependable  architecture  should  have  requirements  associated 
with  it,  which  means  that  certain  dependability  guarantees  should  be  already  reflected  in  the 
architecture  itself.  Then,  testing  is  not  only  to  be  applied  to  the  system  implementation,  but 
also  to  the  architectural  model. 

Thus,  fully  realizing  the  open  architecture  vision  requires  a  new  paradigm  for  test  and 
evaluation.  We  propose  such  a  paradigm  here,  based  on  the  concepts  of  dependability 
contracts,  interchangeable  software  parts,  and  computer-aided  enforcement  of  dependability 
contracts. 

Current  approaches  to  system  development  and  testing  are  more  analogous  to 
individual  craftsmanship  than  they  are  to  modern  concepts  of  mass  production  and 
interchangeable  parts.  Craftsmen  used  to  build  things  by  individually  tuning  mating  parts 
until  they  properly  fit  together.  In  such  a  context,  designs  could  be  relatively  informal  and 
relatively  rough.  In  a  mass  production  environment,  parts  are  built  to  standards  with 
precisely  specified  tolerances,  and  it  is  up  to  the  designer  to  determine  and  verify  the 
tolerances  necessary  to  make  the  design  work  for  any  combination  of  parts  that  meet  the 
specified  tolerances.  An  example  that  illustrates  this  problem  is  the  manufacturing  of  a  rifle 
using  a  set  of  interchangeable  parts.  This  is  different  from  having  parts  that  need  to  be 
crafted  individually.  It  is  necessary  to  evaluate  how  much  variation  is  allowed  to  make 
different  components  and  parts  fit  into  the  rifle.  To  do  this,  it  is  necessary  to  measure 
absolute  sizes  and  construct  the  various  parts  of  the  rifle  with  certain  tolerances.  These 
modular  approaches  have  been  used  in  manufacturing  for  many  years,  but  have  never  been 
successfully  integrated  into  software  engineering  approaches.  Another  example  consists  of 
modern  audio  systems.  There  exist  specific  standards  for  audio  systems  specifying  how 
things  need  to  fit  together  in  order  for  components  from  different  vendors  to  work  together 
effectively.  Standards  for  audio  system  components  can  be  relatively  simple  and  generic 
only  because  the  requirements  for  stereo  systems  are  very  simple.  An  audio  system  is  not 
concerned  about  whether  it  is  playing  a  song  or  the  news.  For  systems  whose  behavior  is 
sensitive  to  the  meaning  of  the  data,  new  types  of  standards  will  be  needed  to  accomplish  a 
similar  function.  These  examples  raise  questions  related  to  the  kinds  of  standards  that  need 
be  considered  to  make  system  components  interchangeable  and  how  such  changes  may 
influence  testing.  The  answers  to  these  questions  should  take  into  account  the  fact  that  we 
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aim  at  testing  pieces  of  the  architecture  versus  standards,  not  versus  (other  pieces  of)  the 
system. 

We  are  seeking  analogous  quality-assurance  techniques  for  systems  involving 
software.  The  fundamental  operation  of  such  an  approach  can  be  outlined  as  follows: 

1 .  System-wide  capabilities  are  characterized  by  a  set  of  dependability  properties  that 
must  hold  in  all  acceptable  system  configurations.  These  properties  comprise  the 
dependability  contract  for  the  system  as  a  whole.  They  become  part  of  a  dependable 
open  architecture  for  the  system  and  serve  as  the  basis  for  system  quality 
assurance.  Dependability  contracts  are  primarily  technical  rather  than  legal 
documents,  and  they  are  intended  to  be  checkable  via  software. 

2.  The  designers  of  the  open  architecture  determine  the  common  structure  of  the 
system  and  develop  the  component-level  dependability  contracts  for  the  subsystems 
and  connectors.  The  common  structure  consists  of  connection  patterns  and 
subsystem  slots  to  which  all  configurations  must  conform. 

3.  The  quality-assurance  team  checks  the  structure  of  the  architecture  and  the 
dependability  contracts  for  subsystems  and  connectors  to  make  sure  they  are  strong 
enough  to  guarantee  the  system-wide  dependability  properties  in  all  possible 
configurations.  This  is  a  one-time  process  that  uses  symbolic  analysis  techniques. 
Assuring  the  feasibility  of  this  step  is  one  of  the  objectives  of  ongoing  research  by  the 
authors. 

4.  The  quality  assurance  team  tests  each  component  (subsystem  and  connector) 
against  its  dependability  contract.  This  is  envisioned  to  be  an  automated  process  to 
enable  sufficient  large  sets  of  test  cases  for  statistically  significant  conclusions  about 
desirable  dependability  levels.  The  cost  for  this  step  is  proportional  to  the  number  of 
components,  and  the  process  must  be  completed  once  for  each  version  of  each 
atomic  component.  Technologies  for  doing  this  are  well  known,  and  many  of  them 
are  used  in  common  practice. 

5.  The  quality-assurance  team  checks  components  for  non-interference.  This  process 
is  computer-aided.  Many  of  the  technologies  for  this  are  well  known,  and  some  of 
them  are  commonly  used.  Some  development  may  be  needed  to  get  a  complete  set. 
This  part  of  the  process  ensures  that  components  that  work  correctly  in  isolation  will 
continue  to  do  so  when  they  are  connected. 

6.  The  assumptions  about  the  operating  environment  on  which  the  architecture 
depends  are  checked  by  runtime  monitoring.  This  can  be  done  using  BIT  (Built-In- 
Test)  technology  that  is  currently  in  use  in  some  DoD  systems.  This  is  recommended 
for  all  reusable  components. 

Figure  6  provides  an  overview  of  the  global  approach.  The  architectural  and  testing 
visions  of  the  proposed  approach  are  described  below: 

■  Architectural  vision 

♦  Consider  an  architecture  as  a  support  system  not  only  for  development  but  also 
for  testing — including  interchangeable  software  parts. 
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♦  Look  at  an  architecture  as  consisting  not  only  of  components,  connections  and 
constraints,  but  also  of  standards,  requirements/capabilities  and  environmental 
assumptions. 

■  Testing  vision 


♦  Relate  testing  to  standards  and  constraints  as  a  means  to  ensure  architecture 
meets  requirements  and  provides  the  needed  capabilities. 

♦  Relate  standards  to  architectural  structures  and  associated  dependability 
requirements. 

♦  Certify  absence  of  interference  between  components  and  the  dependability 
properties  of  interest. 


♦  Check  constraints  on  environment  at  reconfiguration  time. 


♦  The  purpose  is  to  prevent  problems  (such  as  integration  problems).  When 

feasible,  this  is  better  than  detecting  those  problems.  The  approach  should  allow 
for  making  responsibilities  more  visible. 


R  Requirements 

Standard  for  Component  1 

S2  Standard  for  Component  2 

g  Standard  for  connection  between 
“*2  components  1  and  2 

Component  1 
Component  2 


©Testing  Components  vs.  Standards 
(step  4) 

^  Verify  Architecture  vs. 

^  Requirements  &  Standards  (step  3) 
Ensure  non-interference  among 
^  components  (step  5) 


Figure  6.  Overview  of  the  Global  Approach 


The  above  process  is  a  long-term  goal  whose  realization  depends  on  refinement  and 
integration  of  new  technologies  and  processes — especially  those  supporting  steps  3  and  5. 


Full  success  of  the  approach  will  eliminate  the  need  for  integration  testing  after  each 
reconfiguration.  This  is  the  meaning  of  interchangeable  software  components.  We  do  not 
propose  to  eliminate  integration  testing  entirely,  even  in  the  long  term.  The  reason  is  that  all 
analysis  is  relative  to  a  model.  While  the  models  we  use  are  good,  it  is  always  possible  that 
the  existing  implementation  does  not  realize  the  intended  model  completely  precisely.  For 
example,  it  is  possible  that  the  compiler  used  does  not  implement  its  programming  language 
correctly  in  some  rare  cases,  or  that  the  hardware  does  not  perform  its  functions  as 
specified  under  some  rare  conditions.  For  these  reasons,  we  recommend  integration  testing 
for  at  least  one  system  configuration,  e.g.,  the  initial  configuration  to  be  fielded.  Shorter-term 
reductions  in  the  amount  of  testing  needed  after  a  reconfiguration  are  expected  when 
effective  non-interference  checks  eliminate  specific  kinds  of  failures  due  to  integration 
issues. 
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Some  examples  include  interference  due  to  data  or  control  interactions  that  are  not 
allowed  by  the  architecture,  or  due  to  resource  constraints  such  as  limits  on  memory, 
network  bandwidth,  or  computation  time.  An  example  of  existing  technology  that  can 
eliminate  a  specific  type  of  interference  is  architecture-based  schedulability  analysis,  which 
can  guarantee  absence  of  failures  due  to  real-time  constraints  and  computing  resource 
limits. 

Another  related  issue  is  how  to  certify  a  standard.  The  certification  method  should  be 
able  to  satisfy  the  critical  requirements  of  all  architectural  configurations.  Quality  assurance 
and  analysis  techniques  (such  as  model  checking  or  theorem  proving)  could  be  used  for 
such  purposes.  These  techniques  are  well-known,  well-established  and  have  been  used  for 
many  years.  However,  these  techniques  do  not  scale-up  well.  The  reason  is  that  they  have 
traditionally  been  applied  to  program  code,  which  is  a  very  large  artifact.  We  believe  this 
technology  can  be  applied  to  the  architecture  of  a  system  because  the  architecture  is  much 
smaller  than  the  code.  This  is  especially  the  case  if  each  level  of  the  architectural  hierarchy 
can  be  checked  separately.  To  make  this  possible,  the  traditional  concept  of  architecture 
should  be  enhanced,  e.g.,  constraints  and  standards  should  also  be  included. 

Another  issue  is  that  to  check  the  absence  of  interference  between  components, 
static-analysis  techniques  (e.g.,  type  checking,  static  checking,  code  analysis)  will  be 
needed,  since  testing  is  not  enough  for  this  purpose.  This  means  that  reachability  analysis 
techniques  will  necessarily  be  different.  The  underlying  approach  might  be  “large  scale.”  but 
it  does  not  mean  it  needs  to  be  sophisticated  (just  feasible).  Moreover,  if  testing  is 
conducted  against  a  standard,  it  is  possible  to  have  an  automated  testing  oracle.  In  classical 
reliability  techniques,  it  is  possible  to  calculate  the  number  of  test  cases  needed  to  assure 
(with  a  certain  confidence  level)  that  the  system  will  not  experience  more  than  a  given 
number  of  failures  during  a  determined  period  of  time.  For  example,  if  the  system  should  not 
fail  more  than  once  in  N  executions,  the  number  of  test  cases  needed  for  a  confidence  level 
of  1-1/N  is  given  by  Nlog2  N.  (e.g.,  about  20  million  test  cases  are  needed  to  reach  10EXP-6 
assurance). 

Testing  with  respect  to  standards  can  drastically  reduce  the  number  of  test  cases 
needed  because  each  component  can  be  tested  separately,  and  all  possible  combinations 
do  not  need  to  be  checked.  However,  this  source  of  potential  savings  depends  on  effective 
methods  for  carrying  out  steps  3  and  5  above. 

Some  shorter-term  savings  can  be  achieved  by  using  testing  approaches  that  obtain 
information  about  many  different  configurations  based  on  a  single  test  case  run  on  a  single 
configuration.  An  example  is  an  approach  that  tests  every  pair  of  components  that  are 
connected  in  the  architecture  in  at  least  one  system  configuration,  but  not  in  all  possible 
contexts. 

The  major  contributions  and  advantages  of  the  proposed  approach  are: 

■  Ability  to  reduce  the  testing  effort.  The  approach  will  enable  reducing  unnecessary 
testing  on  every  system  change  and  enable  identifying  what  kinds  of  testing  and 
checking  do  need  to  be  repeated  when  something  changes. 

■  Ability  to  limit  the  retesting  scope.  The  approach  will  limit  the  scope  of  retesting  when 
possible.  This  will  involve  a  combination  of  testing  with  other  kind  of  quality- 
assurance  techniques. 
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■  Ability  to  assure  dependability.  The  approach  will  include  methods  for  assuring,  with 
a  single  analysis,  that  all  possible  configurations  that  can  be  generated  in  a  model- 
driven  architecture  will  satisfy  given  dependability  requirements.  Prior  successful 
experience  with  developing  methods  of  this  kind  has  been  demonstrated  by  the  first 
author  of  this  paper  (Berzins,  2000).  These  results  will  be  extended  and  applied  to  fit 
the  requirements  of  the  Navy  open-architecture  initiative. 

Example 

A  simple  example  of  part  of  a  dependable  architecture  is  shown  in  Figure  7.  There 
are  two  component  slots,  one  representing  the  software  driver  for  a  position  sensor,  and  the 
other  a  control  software  module  for  an  autopilot.  There  is  one  connector  that  carries 
information  about  the  current  position  of  the  host  platform.  The  example  shows  just  a 
fragment  of  a  realistic  architecture  as  indicated  by  the  ellipsis  on  the  connections.  In  a 
complete  architecture,  the  position  information  will  also  feed  into  other  systems,  such  as 
tactical  displays  and  weapons  control  systems. 


Overall  dependability  contract 

I  actual. position  -  planned. position  | 

<  navigation. tolerance 


- 

own  craft 
position 

\  position 

autopilot 

[/ 

1 

I  1 

Subsystem  o^iendability  contract 

position. error  <  max. error, 
position. delay  <  max.delay, 

Figure  7.  Example  of  a  Dependable  Architecture  Fragment 

The  figure  shows  a  simplified"^”  partial  description  of  the  dependability  contracts.  The 
overall  purpose  of  the  interconnection  is  to  keep  the  platform  on  course.  This  informal  intent 
is  expressed  as  a  measurable  dependability  property  that  becomes  the  basis  for  the  quality 
assurance  of  this  architectural  fragment.  The  navigation  tolerance  is  a  parameter  of  the 
overall  system  requirements  as  well  as  of  the  architecture.  It  provides  a  partial 
characterization  of  mission  needs  and  a  family  of  system  configurations  that  meet  that  need. 
For  example,  different  types  of  platforms  may  have  different  navigation  tolerances.  Note  that 
this  same  architecture  fragment  is  relevant  to  surface,  subsurface  and  air  platforms.  The 
own-craft-position  subsystem  slot  can  be  filled  by  a  variety  of  sensors,  such  as  GPS,  inertial, 
VOR/DME,  etc.,  and  the  autopilot  subsystem  slot  can  likewise  be  filled  with  components  that 
realize  different  control  algorithms.  The  subsystem  dependability  contract  expresses  part  of 
the  standards  that  any  acceptable  realization  of  the  subsystem  must  meet — by  expressing 
tolerances — for  the  accuracy  of  the  sensor  and  the  allowable  time  delay  between  the  time 
the  platform’s  position  is  measured  and  the  time  the  position  is  delivered  to  the  connector.  It 
is  the  responsibility  of  the  designers  of  the  architecture  to  determine  how  the  values  of  these 


For  example,  for  air  platforms,  the  vertical  navigation  tolerance  can  be  different  than  the  horizontal 
tolerance. 
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subsystem  dependability  parameters  are  derived  from  the  overall  dependability  parameter. 
The  purpose  of  Step  3  in  the  proposed  quality  assurance  process  is  to  check  that  this 
derivation  is  valid  in  the  sense  that  the  system  will  meet  its  requirements  for  any  choice  of 
sensors  that  meets  its  dependability  contract,  as  well  as  for  any  choice  of  control  algorithm 
that  meets  its  dependability  contract  (not  shown).  This  process  depends  on  mathematical 
modeling,  analysis  and  proof  techniques,  some  existing  and  some  to  be  developed. 

The  process  of  reconfiguring  the  system  fragment  in  the  example  would  amount  to 
replacing  the  sensor  and  its  software  driver  with  another  one.  The  quality-assurance 
activities  associated  with  this  would  be  certifying  that  the  new  component  meets  the 
dependability  properties  in  the  own-craft-position  dependability  contract  (Step  4)  and  non¬ 
interference  checks  between  the  new  component  and  the  other  components  in  the  new 
configuration  (Step  5). 

Our  objective  is  to  provide  static  analysis  methods  to  accomplish  Step  5.  If  a 
complete  set  of  such  methods  can  be  provided,  then  integration  testing  will  not  be  needed 
after  such  a  component  replacement.  If  some  but  not  all  of  the  potential  interference  modes 
can  be  ruled  out  by  static  checking,  then  some  integration  testing  will  still  have  to  be 
performed  as  part  of  Step  5,  but  the  scope  of  that  testing  can  be  focused  on  the  failure 
modes  that  are  not  yet  covered  by  static  checks.  We  note  that  although  we  have  been 
mostly  focused  on  replacement  of  software  components,  sometimes,  as  in  this  example,  a 
meaningful  reconfiguration  may  involve  replacement  of  some  hardware  as  well.  In  our 
example,  some  kinds  of  improvements  may  be  possible  by  replacing  just  the  driver  software 
for  a  given  sensor,  but  the  largest  gains  may  come  from  combining  a  new  and  more 
accurate  type  of  position  sensor  with  the  new  software  driver  needed  to  make  the  new 
sensor  fit  the  existing  subsystem  slot  In  the  architecture.  The  goal  is  not  to  change  the 
architecture  when  the  system  is  reconfigured.  In  such  cases,  the  non-interference  tests  may 
include  electrical,  thermal  and  mechanical  considerations  in  addition  to  software 
consideration. 

Our  recommendation  is  to  identify  potential  sources  of  interference  in  detail,  and  to 
develop  specific  quality-assurance  techniques  for  assuring  absence  of  each  type.  These  can 
involve  a  combination  of  static  analysis  checks,  such  as:  data-type  consistency,  lack  of 
unspecified  data  flow,  lack  of  unspecified  control  flow,  conformance  to  power  and  heat  load 
limits,  etc.,  with  conventional  testing  processes.  We  also  note  that  in  some  specific  contexts, 
specialized  efficient  testing  procedures  are  possible,  for  example,  where  dominance 
relations  exist.  For  instance,  in  continuous  domains  it  is  common  that  a  single  worst-case 
test  case  can  expose  all  the  faults  that  any  other  test  case  could  detect. 

The  dependability  contracts  in  the  example  also  have  a  dominance  property:  if  a 
component  has  been  certified  with  respect  to  a  component  with  a  larger  error  tolerance,  it 
will  also  work  for  one  with  a  smaller  error  tolerance,  because  every  possible  behavior  of  the 
more  accurate  component  is  also  a  possible  behavior  of  the  original,  less  accurate 
component.  In  the  example,  a  sensor  with  a  given  max. error  and  max. delay  can  be  replaced 
with  any  other  sensor  that  has  a  smaller  max. error  and  smaller  max.delay  provided  that  the 
new  sensor  also  passes  all  non-interference  checks. 

We  note  that  a  kit  of  available  components  can  be  pre-certified  with  respect  to  Steps 
4  and  5.  This  would  enable  agile  dependable  reconfiguration,  and  perhaps  even  a  capability 
for  on-the-fly  “plug  and  fight.”  The  cost  to  do  Step  4  is  proportional  to  the  number  of 
components,  and  can  economically  be  completed  in  advance.  The  cost  to  do  Step  5 
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depends  on  whether  generic  non-interference  methods  can  be  developed  for  ail  needed 
failure  modes.  In  the  best  case,  it  is  proportional  to  the  number  of  components  and  could  be 
done  in  advance.  If  all-pairs  analysis  is  necessary,  cost  would  be  quadratic  in  the  number  of 
components — making  pre-checking  expensive  but  still  perhaps  feasible  in  advance  if  the 
number  of  components  in  the  reconfigurable  part  is  not  too  big.  In  the  worst  case,  where 
multiple  interactions  may  be  significant,  some  non-interference  checking  may  still  be 
required  after  reconfiguration,  when  the  actual  set  of  components  in  the  new  configuration  is 
known. 

Comparison  with  Reiated  Work 

The  purpose  of  this  section  is  to  provide  a  comprehensive  survey  on  existing 
approaches  for  improving  Quality  Assurance  properties  of  open  and  flexible  architecture- 
based  systems.  It  is  also  an  objective  to  review  existing  works  on  how  testing  is  performed 
in  a  fluid  environment  with  agile  reconfiguration. 

Comparison  with  Navy’s  Approaches 

The  Naval  OA  program  interacts  with  the  OACE,  FORCEnet  and  MOSA  initiatives  in 
different  ways.  As  described  above,  OACE  is  based  on  a  set  of  standards  for  the  computing 
environment  of  surface  ship-centric  systems  specifications;  MOSA  is  an  acquisition  and 
design  approach,  while  FORCEnet  is  a  unifying  concept  for  multiple  architectures  and 
standards  efforts  in  the  Navy.  The  recommended  testing  practices  are  described  by  these 
standards  in  general  terms  and  are  mostly  founded  on  scenario-based  techniques.  For 
example,  OACE  recommends  functional  and  performance  testing  against  specified  system 
requirements,  organized  according  to  test  cases  and  scenarios.  It  defines  the  concept  of 
“virtual  homogeneity”  to  facilitate  testing  by  identifying  groups  of  sub-systems  performing 
similarly.  The  concepts  of  “tree  of  subsystems”  and  “aggregations  of  components”  are  also 
introduced.  Each  aggregation  exists  only  in  a  manageable  number  of  configurations.  A  test 
case  can  be  applied  to  many  configurations  when  there  is  no  (considerable)  interaction 
between  choices  of  configurations.  Schedulability  analysis  is  recommended  for  ensuring  that 
any  configuration  that  the  resource  manager  creates  is  schedulable.  These  are  existing 
attempts  to  reduce  cost  of  testing  by  limiting  flexibility  of  systems  and  to  increase  confidence 
that  a  test  case  provides  useful  information  about  more  than  one  configuration  by  limiting 
possible  sources  of  interference  between  components. 

Our  methodology  aims  at  defining  a  broader  testing  approach  covering  both 
functional  and  non-functional  properties  of  Open  Architecture-based  systems,  with  emphasis 
on  ensuring  dependability  for  all  possible  system  configurations.  Instead  of  seeking  for 
subsystems  performing  similarly  (concept  of  “virtual  homogeneity”),  our  approach  will  use 
architectural  artifacts  and  standards  which  already  define  the  basis  for  all  the  different 
groups  of  subsystems  that  can  be  developed  in  practice.  In  our  context,  “performing 
similarly”  means  “meeting  the  dependability  contract  associated  with  a  subsystem  slot  in  the 
open  architecture.”  Since  our  approach  will  work  at  the  architectural  level,  and  the 
architecture  represents  a  family  of  systems  and  subsystems,  the  concepts  of  “tree  of 
subsystems”  and  “aggregation  of  components”  will  be  also  covered.  The  non-interaction 
between  choices  of  configurations  is  already  covered  by  the  concept  of  non-interference 
defined  in  our  approach.  Schedulability  analysis  is  also  part  of  the  non-interference  notion, 
since  it  will  allow  for  predicting  resource  conflicts  between  tasks  and  processes. 
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Comparison  with  Component-based  Testing 

Component-based  testing  can  be  readily  employed  for  Step  4  of  our  methodology  to 
further  test  a  candidate  software  component  against  the  specific  domain  and  architectural 
standards  of  the  target  system  in  which  it  is  to  be  plugged-in  and  integrated. 

Traditionally,  component-based  testing  is  performed  by  the  component’s  developer 
itself  (e.g.,  through  unit  testing).  It  Is  aimed  at  establishing  the  proper  functioning  of  the 
component  and  at  detecting  possible  failures  early,  i.e.,  ensuring  the  quality  of  the 
component  before  it  is  released.  The  tests  established  by  the  developer  can  rely  not  only  on 
a  complete  documentation  and  knowledge  of  the  component,  but  also  on  the  availability  of 
the  source  code,  and,  thus,  in  general  pursue  some  kind  of  coverage  testing.  Therefore, 
when  applied  by  the  component’s  developer,  this  testing  approach  cannot  address  the 
verification  of  the  component’s  behavior  with  respect  to  the  specifications  of  the  host 
system(s)  in  which  the  component  will  be  later  assembled  (i.e.,  integration  and  system 
testing).  Note,  however,  that  component-based  testing  techniques  are  also  used  by  system 
testers  and  integrators. 

Voas  (1998,  June;  2000,  August)  proposed  a  certification  strategy  for  off-the-shelf 
components  relying  on  black-box  testing,  system-level  fault  injection  and  defense  protection 
through  wrapping.  Black-box  testing  is  a  well-known  testing  technique  used  whenever  the 
source  code  of  a  component  is  not  available,  only  its  interface  specifications.  System  fault 
injection  and  defense  wrapping  are  system-level  approaches  for  integration  testing  and  fault 
containment  that  might  not  be  needed  in  our  approach  if  the  non-interference  property  is 
fully  satisfied. 

Other  approaches  aim  at  making  component’s  data  available  (e.g.,  internal 
behavioral  and  structural  data,  development  data,  etc.)  so  that  the  data  can  assist  the 
testing  process.  The  work  in  Orso,  Harrold  and  Rosenblum  (2001)  defines  an  approach  in 
which  metadata  of  a  component  (describing  both  static  and  dynamic  aspects)  are  available 
throughout  the  entire  component’s  lifecycle.  The  feasibility  of  the  approach  is  demonstrated 
in  the  context  of  component-based  testing,  consisting  of  the  generation  of  self-checking 
code  and  program  slicing.  The  work  in  Whaley,  Martin  and  Lam  (2002)  automatically 
extracts  a  finite-state  machine  model  from  the  interface  of  a  software  component,  which  can 
be  delivered  along  with  the  component  itself  for  testing  purposes.  Off-the-shelf  (OTS) 
components  are  usually  acquired  as  black-box  code  without  access  to  data  that  might  be 
necessary  for  testing.  Salles,  Rodriguez,  Fabre  and  Arlat  (1999)  developed  a  framework  for 
integration  testing  of  OTS  real-time  operating  systems  (RTOS).  Information  needed  for 
testing  is  obtained  through  reflective  techniques  implemented  in  an  additional  software 
module  added  to  the  OTS  component.  A  fault-injection  methodology  is  used  to  verify  that 
the  behavior  of  the  integrated  OTS  component  does  not  impact  the  dependability  of  the 
system. 

Bertolino  and  Polini  (2003)  recognized  the  importance  of  testing  a  software 
component  in  its  deployment  environment  (i.e.,  the  target  system).  They  developed  a 
framework  that  supports  functional  testing  of  a  software  component  with  respect  to 
customer’s  specification — which  also  provides  a  simple  way  to  enclose  the  developer’s  test 
suites  which  can  be  re-executed  by  the  customer.  The  customer  is  thus  provided  with  both  a 
technique  to  specify  a  deployment  test  suite  early  and  an  environment  for  running  and 
reusing  the  specified  tests  on  any  component  implementation.  There  is  a  complete 
decoupling  between  the  tests’  specification  and  component  implementation.  The  approach 
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requires  the  customer  to  have  a  complete  specification  of  the  component  to  be  incorporated 
into  a  system. 


In  the  formal  verification  domain,  there  has  been  a  long  history  of  research  on 
verification  of  systems  with  modular  structure.  A  key  idea  (Lamport,  1983;  Kupferman  & 
Vardi,  1997;  Henzinger,  Qadeer  &  Rajamani,  1998)  in  modular  verification  is  the  assume- 
guarantee  paradigm:  a  module  should  guarantee  to  have  the  desired  behavior  once  the 
environment  with  which  the  module  is  interacting  has  the  assumed  behavior.  There  have 
been  a  variety  of  implementations  for  this  idea  (see,  e.g.,  Grumberg  &  Long,  1994;  Alur  et 
al.,  1998;  Pasareanu,  Dwyer  &  Huth,  1999;  Dingel,  2003;  Chaki,  Clarke,  Groce,  Jha  &  Veith, 
2003;  Xie  &  Browne,  2003).  The  key  issue  with  the  assume-guarantee  style  reasoning  is 
how  to  obtain  assumptions  about  the  environment.  Giannakopoulou  et  al.  (Giannakopoulou, 
Pasareanu  &  Barringer,  2002;  Giannakopoulou,  Pasareanu  &  Cobleigh,  2004)  introduced  a 
novel  approach  to  generate  assumptions  that  characterize  exactly  the  environment  in  which 
a  component  satisfies  its  property.  Their  idea  is  based  on  a  purely  formal  verification 
technique  (model-checking).  Fisleretal.  (Fisler&  Krishnamurthi,  2001;  Li,  Krishnamurthi  & 
Fisler,  2002)  introduced  a  similar  idea  of  deducing  a  model-checking  condition  for  extension 
features  from  the  base  feature  for  model-checking,  feature-oriented  software  designs.  This 
approach  is  not  applicable  to  component-based  systems  where  unspecified  components 
exist.  This  work  differs  from  related  work  like  Xie  and  Dang  (2004),  in  which  an  automata- 
theoretic  approach  is  used  to  solve  a  similar  LTL  model-checking  problem. 

In  the  past  decade,  there  has  also  been  significant  research  on  combining  model¬ 
checking  and  testing  techniques  for  system  verification,  which  can  be  grouped  into  a 
broader  class  of  techniques  called  specification-based  testing.  Many  of  the  studies  utilize 
model-checkers’  ability  of  generating  counter-examples  from  a  system’s  specification  to 
produce  test  cases  against  an  implementation  (Callahan,  Schneider  &  Easterbrook,  1996; 
Floizmann,  1997,  May;  Engels,  Feijs  &  Mauw,  1997;  Gargantini  &  Fleitmeyer,  1999; 
Ammann,  Black,  &  Majurski,  1998;  Black,  Okun,  &  Yesha,  2000).  Peled  et.  al.  (Peled,  Vardi 
&  Yannakakis  1999:  Groce,  Peled  &  Yannakakis,  2002;  Peled,  2003)  studied  the  issue  of 
checking  a  black-box  against  a  temporal  property  (called  black-box  checking).  The  research 
focuses  on  how  to  efficiently  establish  abstract  models  for  black-box  testing  and  on  how  to 
define  properties  (e.g.,  LTL  formula)  about  the  black-box  components. 

Comparison  with  Runtime  Software  Reconfiguration 

For  an  important  class  of  safety-  and  mission-critical  software  systems,  such  as  air 
traffic  control,  telephone  switching,  and  high-availability  public  information  systems,  shutting 
down  and  restarting  the  system  for  upgrades  incurs  unacceptable  delays,  increased  cost, 
and  risk.  Support  for  runtime  modification  is  a  key  aspect  of  these  systems.  In  our 
methodology,  a  reconfigured  set  of  components  can  be  seen  as  a  particular  configuration  of 
a  system  architecture.  Since  our  approach  aims  at  guaranteeing  dependability  properties  for 
the  family  of  systems  and  configurations  represented  by  the  architecture,  the  proposed 
testing  approach  should  be  able  to  provide  assurance  in  presence  of  runtime  reconfiguration 
for  at  least  a  certain  number  of  properties  (e.g.,  non-interference). 

There  are  a  wide  variety  of  techniques  for  supporting  runtime  software  change. 

Some  of  the  most  popular  techniques  are  based  on  Dynamic  Software  Architectures 
(Oreizy,  2007).  Several  research  projects  have  addressed  these  issues,  such  as  Self- 
Adaptive,  Healing  Architectures  (ArchShell,  2007),  or  Dynamic  Wright  (Allen,  Douence  & 
Garlan,  1998,  April).  Gupta,  Jalote  and  Barua  (1996,  February)  describe  an  approach  to 
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modeling  changes  at  the  statement  and  procedure  levels  for  a  simple  imperative 
programming  language.  Many  dynamic  programming  languages,  such  as  Lisp,  Smalltalk, 
and  Haskel  (Peterson,  Hudak  &  Ling,  1997,  July)  have  supported  runtime  software  change 
for  decades.  Dynamic  linking  mechanisms  and  libraries  have  been  available  in  operating 
systems  such  as  UNIX,  Microsoft  Windows,  and  the  Apple  Macintosh  for  some  time.  New 
approaches  to  dynamic  linking  (Franz,  1997,  March)  hope  to  significantly  reduce  the  runtime 
performance  overhead  associated  with  using  such  mechanisms.  Dynamic  Object 
Technology,  such  as  CORBA  (Object  Management  Group,  1996,  July)  and  COM 
(Brockschmidt,  1994)  support  the  runtime  locations,  loading,  and  binding  of  software  objects 
or  components. 

Service-oriented  architectures  (SOAs)  typically  have  a  dynamic  nature,  given  by  the 
runtime  detection  of  components  through  registry  services  and  subsequent  dynamic  binding. 
The  work  in  Baresi,  Heckel,  Thone,  and  Varro  (2006)  defines  a  refinement  relation  from  a 
generic  style  of  component-based  systems  to  the  SOA  style  based  on  the  use  of  graph 
transformation  systems  as  models  of  architectural  styles  at  different  levels  of  platform 
abstraction  (which  represent  reconfiguration  and  communication  scenarios  as  graph 
transformation  sequences).  Besides  the  many  proposals  for  Architecture  Description 
Languages  (ADLs),  like  Rapide  (Luckham  et  al.,  1995;  Oreizy,  1996,  August;  Oreizy, 
Medvidovic  &  Taylor,  1998,  April),  Wright  (Allen,  1997;  Allen,  Douence  &  Garlan,  1997, 
September;  Allen,  Douence  &  Garlan,  1998,  April;  Allen,  Douence  &  Garlan,  1998),  Darwin 
(Magee,  Dulay,  Eisenbach  &  Kramer,  1995;  Kramer  &  Magee,  1998)  or  C2  (Medvidovic, 
1996,  October;  Oreizy,  Medvidovic  &  Taylor,  1998,  April),  we  must  mention  those 
approaches  that  exploit  graph  transformation  (Hirsh,  2003;  Hirsh  &  Montanari,  2001,  August; 
Metayer,  1996,  October;  Taentzer,  Goedicke  &  Meyer,  2000;  Wermelinger  &  Fiadeiro,  2002; 
Baresi,  Heckel,  Thone,  &  Varro,  2003;  Gonczy,  2006)  to  reason  about  the  consistency  of 
reconfiguration  operations  and  interaction  of  components  with  respect  to  structural 
constraints.  Le  Metayer  (1996,  October)  describes  architectures  by  graphs  and  the  valid 
graphs  of  an  architectural  style  by  a  graph  grammar.  Reconfiguration  is  described  by 
conditional  graph-rewriting  rules.  He  uses  static-type  checking  to  prove  that  the  rewriting 
rules  are  consistent  with  the  respective  style.  The  graphs  represent  computational  entities 
but  not  connectors,  specifications,  or  other  resources.  Wermelinger  and  Fiadeiro  (2002) 
provide  an  algebraic  framework  based  on  Category  theory  in  which  architectures  are 
represented  as  graphs  of  Community  programs  and  superpositions.  Dynamic 
reconfigurations  are  specified  by  graph  transformation  rules  over  architecture  instances. 

Both  styles  and  rules  are  used  for  modeling  domain-specific  restrictions  rather  than  the 
underlying  platform.  Consequently,  they  do  not  deal  with  refinement  relationships  between 
different  levels  of  platform  abstraction.  Hirsch  (2003)  uses  hypergraphs  to  represent 
architectures  and  hyperedge  replacement  grammars  to  define  the  valid  architectures  of  an 
architectural  style.  Furthermore,  he  uses  graph  transformation  rules  to  specify  runtime 
interactions  among  components,  reconfigurations,  and  mobility.  In  the  CHAM  approach 
(Inverardi  &  Wolf,  1995,  April),  architectural  reconfiguration  is  studied  in  terms  of  molecules 
and  reactions,  and  the  proposals  that  represent  architectural  styles  by  means  of  graph 
grammars  (Hirsh  &  Montanari,  2001,  August;  Metayer,  1996,  October;  Taentzer,  Goedicke  & 
Meyer,  2000;  Wermelinger  &  Fiadeiro,  2000,  March)  and  reason  on  changes  and  evolution 
with  respect  to  structural  constraints.  Some  of  these  approaches  use  a  graph  grammar  to 
specify  the  class  of  admissible  configurations  of  the  style.  Graph  transformation  rules  model 
only  dynamic  aspects  like  evolution  and  reconfiguration.  The  advantage  is  that  a  declarative 
specification  is  more  abstract  and  easier  to  understand,  even  if  constructive/operational 
ones  are  better  for  analysis  and  tools.  The  use  of  graph-transformation  techniques  to 
capture  dynamic  semantics  of  models  has  also  been  inspired  by  work  proposed  by  Engels, 
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Hausmann,  Heckel  and  St.  Sauer  (2000)  under  the  name  of  dynamic  meta-modeling.  That 
approach  extends  metamodels — re-defining  the  abstract  syntax  of  a  modeling  language  like 
UML  by  using  graph-transformation  rules  that  allow  for  describing  changes  to  object  graphs 
and  represent  the  states  of  a  model. 

Grammar-oriented  Programming  (GOP)  and  Grammar-oriented  Object  Design 
(GOOD)  (GOOD,  2002)  are  based  on  designing  and  creating  a  domain-specific 
programming  language  (DSL)  for  a  specific  business  domain.  GOOD  can  be  used  to  drive 
the  execution  of  the  application,  or  it  can  be  used  to  embed  the  declarative  processing  logic 
of  a  context-aware  component  (CAC)  or  context-aware  service  (CAS)  (Arsanjani,  Curbera,  & 
Mukhi,  2004).  GOOD  is  a  method  for  creating  and  maintaining  dynamically  reconfigurable 
software  architectures  driven  by  business-process  architectures.  The  business  compiler  was 
used  to  capture  business  processes  within  real-time  workshops  for  various  lines  of  business 
and  create  an  executable  simulation  of  the  processes  used.  Instead  of  using  one  DSL  for 
the  entire  programming  activity,  GOOD  suggests  the  combination  of  defining  domain- 
specific  behavioral  semantics  in  conjunction  with  the  use  of  more  traditional,  general 
purpose  programming  languages. 

The  use  of  model-checking  techniques  for  verifying  software  architectures  has  been 
thoroughly  studied.  For  example,  vUML  (Lilius  &  Paltor,  1999,  October),  veriUML  (Compton, 
Gurevich,  Huggins  &  Shen,  2000),  JACK  (Gnesi,  Latella  &  Massink,  1999),  and  HUGO 
(Schafer,  Knapp  &  Merz,  2001)  support  the  validation  of  distributed  systems  (where  each 
statechart  describes  a  component),  but  do  not  support  complex  communication  paradigms. 
These  works  study  static  systems  whose  topology  cannot  vary  at  runtime.  Similarly,  Garlan 
Khersonsky  and  Kim  (2003,  May)  and  the  researchers  involved  in  the  Cadena  project 
(Hatcliff,  Deng,  Dwyer,  Jung  &  Ranganath,  2003,  May)  applied  model-checking  techniques 
to  analyze  specific  architectures  with  a  fixed  topology  based  on  the  publish/subscribe 
paradigm.  A  formal  approach  that  considers  refinement  of  dynamic  reconfiguration  can  be 
found  in  Bolusset  and  Oquendo  (2002).  The  approach  is  targeted  on  the  translation  from 
one  ADL  to  another  rather  than  on  the  refinement  between  architectural  styles.  Cherchago 
and  Heckel  (2004)  describe  the  application  of  graph  transformations  in  the  runtime  matching 
of  behavioral  Web  service  specifications.  In  Heckel  and  Mariani  (2005),  the  conformance 
testing  of  Web  services  is  based  on  graph  transformations,  focusing  on  the  automated  test- 
case  generation.  The  work  of  Bertolino  and  Polini  (2006)  utilizes  the  benefits  of  these 
approaches  and  defines  fault-tolerant  algorithms  incorporated  into  appropriate 
reconfiguration  mechanisms  for  modeling  reliable  message  delivery  by  graph-transformation 
rules  in  SOA.  Graph  transformation  is  used  as  a  specification  technique  for  dynamic 
architectural  reconfigurations  in  Wermelinger  and  Fiadeiro  (2002),  using  the  algebraic 
framework  Community.  Hirsch  uses  graph  transformations  over  hypergraphs  (2003)  to 
specify  runtime  interactions  among  components,  reconfigurations,  and  mobility  in  a  given 
architectural  style.  A  profile  for  reliability  was  designed  for  J2EE  applications  in  Rodrigues, 
Roberts,  Emmerich  and  Skene  (2004).  In  Zheng,  Jun  and  Yan  (2005),  a  pattern-based 
specification  and  runtime  validation  approach  is  presented  for  interaction  properties  of  web 
services  using  a  semantic  web  rule  language  (SWRL).  GROOVE  (Graphs  for  Object- 
oriented  Verification,  2007)  is  a  project  centered  around  the  use  of  simple  graphs  for 
modeling  the  design-time,  compile-time,  and  runtime  structure  of  object-oriented  systems;  it 
also  focuses  on  graph  transformations  as  a  basis  for  model  transformation  and  operational 
semantics.  This  entails  a  formal  foundation  for  model  transformation  and  dynamic 
semantics,  and  the  ability  to  verify  model  transformation  and  dynamic  semantics  through  an 
(automatic)  analysis  of  the  resulting  graph  transformation  systems — for  instance  using 
model  checking. 
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The  techniques  described  above  such  as  model  checking  and  graph  transformations 
can  benefit  several  steps  of  our  methodology.  In  Step  2  these  techniques  can  be  used  to 
derive  dependability  contracts  addressing  reconfiguration,  topology  and  connections 
properties  and  constraints  at  the  system  and  subsystem  levels.  In  Steps  3  and  4,  these 
techniques  can  be  used  in  combination  with  other  symbolic  analysis  techniques  and  testing 
techniques  for  verifying  the  structure  of  the  architecture  and  the  dependability  contracts. 
These  techniques  can  also  provide  useful  information  about  the  various  sources  of 
interference  between  the  target  components  and  the  host  system,  and  can  help  determine 
suitable  and  alternate  approaches  to  avoid  those  interference  sources. 

However,  designers  have  traditionally  sought  alternatives  to  runtime  change, 
especially  for  safety-critical  applications  such  as  combat  systems.  Several  reasons  account 
for  this: 

1 .  It  is  usually  avoidable.  Runtime  change  is  not  a  critical  aspect  of  many  software 
systems,  and  several  techniques  have  been  devised  to  circumvent  the  need  for 
runtime  change  altogether.  Regularly  scheduled  downtimes,  functional  redundancy 
or  clustering,  and  manual  overrides  are  all  examples  of  such  techniques. 

2.  It  increases  risk.  System  integrity,  reliability,  and  robustness  are  more  difficult  to 
ensure  in  light  of  runtime  change. 

3.  It  increases  cost.  There  is  typically  a  marked  performance  overhead  associated  with 
supporting  runtime  change.  Additionally,  few  techniques  have  limited  expertise;  and 
a  lack  of  proven  techniques  for  supporting  runtime  change  exasperates  engineering 
costs. 

Although  “plug  and  fight”  has  been  articulated  as  a  goal,  in  the  near  term 
reconfiguration  is  likely  to  be  more  constrained  and  less  agile  due  to  weapons  certification 
and  doctrine  issues. 

Conclusion 

This  paper  explores  methods  for  test  and  evaluation  of  flexible  systems  with  open 
architectures,  and  proposes  an  approach  for  substantially  reducing  the  amount  of  testing 
necessary  for  dependable  reconfigurable  systems.  The  approach  involves  augmenting  open 
architectures  with  measurable  dependability  properties  associated  with  the  system  as  a 
whole  as  well  as  dependability  properties  associated  with  slots  for  replaceable  subsystems. 
It  also  involves  augmenting  testing  with  other  kinds  of  quality-assurance  methods.  These 
additional  methods  include  static  checks  for  non-interference  properties.  The  purpose  of 
these  checks  is  to  ensure  that  components  that  work  correctly  in  isolation  will  continue  to  do 
so  in  the  context  of  a  given  dependable  open  architecture.  In  the  long  term,  this  approach 
should  eliminate  the  need  for  integration  testing  after  each  reconfiguration,  and  in  the  short 
to  medium  term,  it  should  substantially  reduce  the  amount  of  integration  testing  required 
after  reconfiguration. 
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Abstract 

Federal  Government  agencies  convert  in-house  positions  to  contractor  positions  via 
the  A-76  process  in  order  to  save  money.  During  an  A-76  conversion,  stable  future  labor 
cost  growth  is  assumed.  This  assumption  is  faulty  in  cases  wherein  the  contractor 
workforce  subsequently  unionizes.  Unionization  may  lead  to  unanticipated  increases  in 
cost,  threatening  the  savings  projected  during  the  A-76  process.  This  study  seeks  to:  1) 
compare  the  rate  of  labor-cost  growth  for  military,  civilian,  and  contract  employees  and  2) 
compare  current  labor  costs  for  a  sample  outsourced  activity  (fuels)  to  labor  costs  for  the 
military  requirement  based  on  manpower  standards.  The  study  finds  that  overall,  annual 
collective  bargaining  agreement  (CBA)  wages  increases  were  typically  1-2%  higher  than 
Service  Contract  Act  (SCA)  wages  increases.  However,  we  found  no  evidence  that 
contractors  performed  functions  in  our  sample  functional  area  at  a  higher  cost  than  in- 
house.  The  actual  cost  of  contractors  averaged  -40%  less  than  the  cost  derived  from  the 
military  manpower  standard.  The  actual  in-house  cost  averaged  over  20%  higher  than  the 
cost  derived  from  the  manpower  standard.  Our  results  indicate  that  for  the  fuels  functional 
area,  A-76  actions  remain  economically  advantageous  to  the  government,  despite  increases 
in  contract  labor  cost. 

Introduction 

In  1966,  the  US  Government  began  using  the  A-76  process  to  evaluate 
governmental  activities  as  candidates  for  outsourcing  (0MB,  1983,  August  4).  The  A-76 
process  provides  a  roadmap  for  government  to  convert  work  from  internal  (military  in  our 
study)  performance  to  either  contractor  support  or  a  redefined  in-house  performance 
organization.  The  intent  of  the  A-76  process  is  to  save  money  while  freeing  military 
personnel  to  perform  inherently  governmental  functions.  The  A-76  process  requires  the  use 
of  estimates  of  future  cost  growth  when  evaluating  cost  not  just  in  the  current  year,  but  in  the 
out-years  as  well.  These  estimates  are  based  on  assumptions  of  stable  future  cost  growth. 

Several  potential  frustrations  may  develop  for  commanders  in  the  post-A76 
environment.  These  frustrations  include  a  perceived  loss  of  flexibility  in  mission  support 
when  moving  from  military  to  contractor  support.  Further,  there  may  be  frustration  with 
“must  pay”  contract  cost  increases  stemming  from  an  increase  in  union-negotiated  collective 
bargaining  agreements  (CBAs)  and  the  perception  that  contractors  have  no  incentive  to 
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control  CBA  cost  growth  because  the  increase  is  passed  on  to  the  government  agency  as  a 
100%  pass-through. 


The  impact  of  increasing  labor  costs  for  service  contracts  is  felt  Air  Force-wide. 
Activities  are  competitively  sourced  for  various  reasons,  but  cost  savings  are  a  primary 
driver.  When  the  cost  of  the  contracted  services  expands  beyond  expectations,  then  only 
two  options  exist  for  the  government:  reduce  the  level  of  service  or  take  funding  away  from 
other  requirements.  Both  of  these  actions  have  a  negative  impact  on  day-to-day  operations. 

Loss  of  flexibility  coupled  with  ever-increasing  contract  cost  has  caused  some 
commanders  to  ask  whether  or  not  the  A-76  evaluation  process  adequately  considers  future 
labor  cost  growth  in  the  decision  to  outsource. 

Wage  Rate  History 

Wage  rates  in  contracted-out  functions  are  controlled  by  the  Service  Contract  Act 
(SCA).  The  SCA’s  purpose  is  to  protect  the  wages  and  benefits  of  service  contract 
employees.  41USC351  provides  required  provisions  for  public  contracts  in  excess  of 
$2500.  The  Secretary  of  Labor  creates  wage  determinations  to  establish  minimum  wages 
for  various  categories  of  workers  in  a  location.  41 USC351  (a)  requires  the  contractor  to 
provide  wages  and  fringe  benefits  at  least  equal  to  the  amounts  established  in  the 
Department  of  Labor  wage  determination. 

The  Fair  Labor  Standards  Act  and  Service  Contract  Act,  ‘Price  Adjustment  (Multiple 
Year  and  Option  Contracts)”  clause  incorporates  this  law  into  government  contracts  {FAR, 
2007,  52.222-43).  It  applies  to  both  contracts  covered  under  the  SCA  as  well  as  those 
under  which  the  workforce  is  unionized,  and  it  operates  under  a  CBA.  The  contractor’s 
employees  are  protected  by  these  Acts  as  their  pay  and  fringe  benefits  cannot  fall  below  the 
wage  determination  of  the  Department  of  Labor.  The  contractor  is  protected  by  these  acts 
because  the  increases  in  pay  and  fringe  benefits  are  pass-through  costs  to  the  government. 
Thus,  the  contractor  is  not  subject  to  financial  ruin  due  to  increasing  labor/fringe  benefit 
rates.  The  government  takes  on  the  financial  burden  of  these  increases  to,  in  effect,  protect 
the  contractor’s  employees  (Lear  Siegler  Services).  This  creates  a  financial  obligation  to  the 
contracting  entity  which  that  entity  has  little  input  or  control  over.  In  SCA  actions,  the 
increases  reflect  the  rise  in  wages  in  the  local  area  per  the  wage  determination,  and  are, 
therefore,  seen  as  somewhat  predictable.  CBA  negotiations  between  the  contractor  and  the 
employees’  union  are  not  bound  to  such  limitations,  and  are  seen  as  less  predictable. 

Two  additional  Federai  Acquisition  Reguiation  (FAR)  citations  are  very  relevant. 

FAR  22.1008-2  (b)  mandates  that  wage  determinations  and  CBAs  carry  over  even  if  there  is 
a  change  in  contractor  due  to  the  contract  being  reawarded.  The  intent  here  is  clear: 
because  the  competing  bidders  are  bound  to  the  same  labor  rates  as  the  incumbent,  they 
cannot  develop  a  bid  advantage  solely  by  discounting  employees’  wages.  The  effect  of  this, 
however,  may  be  to  reduce  the  incentive  of  the  incumbent  to  drive  a  hard  bargain  in 
negotiations  with  the  union.  For  one,  they  do  not  actually  pay  the  Increases,  and  secondly, 
their  competition  gains  no  advantage  from  the  negotiated  increases. 

Finally,  FAR  22.1002-3  and  FAR  22.1021  do  provide  avenues  for  the  government  to 
ensure  that  the  negotiations  between  the  contractor  and  union  are  legitimate.  FAR  22.1021 
allows  the  contracting  officer  to  request  a  hearing  with  the  Department  of  Labor  (DoL)  to 
determine  if  the  negotiated  CBA  rates  are  reasonable.  The  CBA  will  not  be  applied  if: 
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The  Secretary  of  Labor  determines — (1)  after  a  hearing,  that  the  wage  and 
fringe  benefits  are  substantially  at  variance  with  those  which  prevail  for 
services  of  a  similar  character  in  the  locality,  or  (2)  that  the  wages  and  fringe 
benefits  are  not  the  result  of  arm’s  length  negotiations.  {FAR,  2007,  22.1002- 
3  (a)(1)  (2)) 

The  question  of  how  much  the  wage  and  fringe  benefit  rates  of  collective  bargaining 
agreements  vary  from  the  SCA  rates  raised  by  this  FAR  reference  is  at  the  heart  of  this 


research. 


Three  previous  studies  have  addressed  labor-cost  growth  issues  and  were 
referenced  in  this  study.  The  first  was  conducted  by  the  Center  for  Naval  Analyses  in  2001 , 
entitled,  “Long-run  Costs  and  Performance  Effects  of  Competitive  Sourcing”  (Rosenblum, 
Coast  &  Smallwood,  2001).  The  second  study  entitled,  “Personnel  Savings  in  Competitively 
Sourced  DoD  Activities:  Are  they  Real?  Will  They  Last?”  was  conducted  by  RAND  in  2000 
(Gates  &  Robbert).  The  third  study  referenced  was  conducted  by  Dr.  Roger  Golden  and 
published  in  his  dissertation  entitled,  “Cost  Trends  on  Defense  Commissary  Service 
Contracts”  (1999). 

When  the  government  converts  work  to  contractor  performance  through  the  A-76 
process,  stable  future  labor  cost  growth  is  assumed.  This  assumption  is  faulty  in  cases 
wherein  the  contractor  workforce  unionizes.  In  addition,  the  assumption  may  lead  to 
unanticipated  increases  in  cost,  which  could  invalidate  savings  projected  during  the  A-76 
process.  Establishing  a  better  method  to  compare  future  labor  cost  will  ensure  the  more 
cost-effective  organization,  over  the  long  term,  performs  the  service. 

The  ultimate  aim  of  this  research  stream  is  to  develop  an  additional  cost  growth 
factor  to  consider  in  the  A-76  process.  This  study  focuses  on  answering  two  fundamental 
questions  necessary  to  develop  such  a  factor:  Do  CBAs  lead  to  faster  wage  growth?  Do 
CBAs  end  up  costing  the  government  more  than  military  performance  of  the  function? 

Objectives 

1 .  Compare  cost  growth  in  labor  rates  of  military,  civilian.  Service  Contract  Act,  and 

collective  bargaining  agreement  employees. 

2.  Compare  current  labor  costs  for  an  A-76-affected  activity  to  labor  costs  for  the 

military  requirement  based  on  manpower  standards. 

The  Sample:  Air  Education  and  Training  Command  (AETC) 

The  Air  Force  Air  Education  and  Training  Command  (AETC)  administers  many 
contracts  with  industry  for  commercial  functions.  AETC  has  conducted  A-76  competitive 
sourcing  studies  over  a  long  period  of  time,  with  many  being  completed  in  the  1980’s  and 
1990’s. 

The  current  challenge  to  AETC  is  that  the  government  is  required  to  pay  the 
contractor  for  wage  increases,  whether  they  originate  from  a  revised  wage  determination 
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under  the  Service  Contract  Act  (SCA)  or  a  renegotiation  of  a  collective  bargaining 
agreement  (CBA).  While  the  SCA  is  considered  a  fairly  stable  and  predictable  cost  growth, 
the  CBA  growth  is  viewed  as  less  predictable.  For  an  organization  with  a  defined  budget, 
being  required  to  fund  unpredictable  labor  cost  requirements  can  have  negative  impacts. 

AETC’s  mission  has  led  to  frequent  use  of  the  A-76  process  to  competitively  source 
commercial  activities.  Based  on  these  factors,  the  study  team  selected  AETC  bases  as  the 
sample  for  this  study. 

Methodology 

Assumptions 

1 .  Direct  comparison  of  base  pay  rates  between  military,  civilian,  SCA,  and  CBA 
employees  Is  acceptable  as  a  method  of  comparing  labor  cost  growth.  Because  the 
composite  factors  for  military  pay  contain  costs  not  obtainable  for  civilian,  SCA  and 
CBA  rates,  the  base  pay  rate  is  the  most  accurate  measure  for  comparison  between 
the  contracted  and  in-house  pay  growth. 

2.  Time  is  not  a  factor  in  calculating  labor  cost  growth  rates  as  the  comparison  between 
rates  covers  the  same  time  periods  and,  therefore,  the  time/inflation  costs  are  the 
same. 

3.  SCA  base  pay  rates  plus  health  and  welfare  (H&W)  costs  are  directly  comparable  to 
CBA  base  pay  rates  plus  H&W  costs.  This  assumption  is  necessary  to  allow  for 
direct  comparison  of  the  two  rates,  although  they  may  not  always  mean  exactly  the 
same  thing  for  each  CBA. 

4.  Overhead  costs  for  Civilian,  SCA  and  CBA  positions  are  assumed  to  be  relatively 
consistent,  allowing  comparison  of  wage  increases  without  incorporating  applicable 
overhead  costs.  This  assumption  is  needed  to  conduct  timely  analysis. 

5.  The  manpower  standard  for  Fuels  can  be  used  as  an  independent  comparison  factor 
for  the  fuels  function.  The  standard  is  not  being  used  as  a  manpower  adjustment 
tool  as  it  is  designed. 

Objective  1 :  Cost  Growth  of  Labor  Rates 

The  first  objective  is  to  compare  the  labor  cost  growth  in  the  various  types  of  pay 
structures:  military,  civilian,  contractor  under  SCA,  and  contractor  under  CBA.  The  purpose 
of  this  comparison  was  to  determine  if  CBA  wages  increased  at  a  higher  rate  than  SCA 
wages  and  to  compare  that  wage  growth  to  military  and  civilian  wage  growth.  The 
development  of  this  comparison  involved  several  steps. 

The  analysis  was  conducted  by  location  because  the  pay  scales  and  wage  increases 
for  SCA,  and  CBA  employees  varied  by  location.  The  first  step  was  to  identify  AETC  bases 
that  had  multiple  CBAs.  This  is  important  because  while  the  other  three  pay  types  are 
widespread,  there  are  a  limited  number  of  CBAs.  Some  installations  had  just  one  CBA. 
Requiring  a  location  to  have  multiple  CBAs  in  order  to  be  included  is  a  precaution  to  prevent 
one  aberrant  CBA  from  skewing  the  location’s  comparison.  After  that,  the  pay  increases 
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were  calculated  for  each  position  and  compared.  The  wages  were  also  combined  to 
analyze  the  wage  growth  for  each  type  of  pay  over  all  of  AETC. 


The  analysis  of  wage  increases  was  conducted  on  base-pay  rates  and  composite- 
pay  rates.  For  base-pay  rates,  there  are  no  fringe  benefits  (e.g.,  health  and  welfare  costs) 
or  overhead  (e.g.,  FICA,  SUDA,  FUDA,  etc.)  costs  included  in  the  wages.  For  military 
wages,  the  base-pay  rates  did  not  include  BAFI,  BAS,  or  any  other  fringe  costs.  For 
composite-pay  rates,  fringe  benefits  are  included  for  all  pay  types,  but  the  overhead  costs 
are  only  included  in  the  military  pay  rates.  The  overhead  costs  would  be  the  same  for  SCA 
and  CBA-determined  wages  because  the  contractor  employing  each  position  would  not 


change. 


Objective  2:  Contractor  vs.  Military  Cost  Growth 

The  second  objective  was  to  compare  actual  contractor  costs  and  actual 
military/civilian  costs  for  a  specific  function  to  the  manpower  standard  cost  for  that  function. 
Although  manpower  standards  are  not  recommended  for  A-76  competitions  or  reverse  A-76 
actions,  this  study  used  the  manpower  standard  to  contrast  how  in-house  and  contractor 
functions  related  to  the  standard.  The  manpower  standard  was  used  because  manpower 
data  was  usually  unavailable,  outside  of  a  limited  timeframe,  after  the  completion  of  an  A-76 
competition.  The  end  result  of  this  objective  is  a  comparison  of  the  actual  costs  of  the 
function  performed  by  military,  civilian,  and  CBA  employees  to  a  calculated  cost  using  the 
manpower  standard. 

The  first  step  was  to  identify  a  function  that,  within  AETC,  is  performed  in-house  at 
some  locations  while  contracted  out  at  others  under  collective  bargaining  agreements.  The 
function  had  to  have  a  relatively  stable  workload  that  could  be  easily  quantified.  The  fuels 
function  at  AETC  bases  met  these  requirements,  and  as  such  was  selected  as  a 
representative  function  for  this  study. 

The  next  step  was  to  calculate  the  manpower  standards  based  on  the  Fuels 
Management  Manpower  Standard  outlined  in  AFMS  41  DA.  The  resulting  manpower 
standards  for  each  base  were  then  converted  to  positions  as  described  in  AFMS  41  DA. 
These  positions  were  next  converted  to  total  cost.  After  that,  the  actual  costs  of  the  fuel 
functions  at  each  base  were  compared  with  their  respective  manpower  standard  costs.  For 
in-house  costs,  the  assigned  personnel  numbers  obtained  from  the  AFPC  Authorized  and 
Assigned  database  were  matched  to  position-costs  to  compute  the  actual  costs  of  the  in- 
house  organization.  For  contracted-out  costs,  the  actual  contract  costs  were  used. 

Finally,  the  cost  ratios  of  actual  cost  to  projected  costs  using  the  manpower  standard 
were  calculated.  These  results  were  presented  numerically  and  graphically. 

Analysis 

Objective  1 :  Cost  Growth  of  Labor  Rates 

Of  the  AETC  bases,  seven  bases  provided  data  on  three  or  more  collective 
bargaining  agreements:  Columbus,  Keesler,  Lackland,  Laughlin,  Randolph,  Sheppard,  and 
Tyndall.  Personnel  at  these  locations  provided  the  actual  wage  tables  from  the  CBAs. 
Military  base-pay  and  composite-pay  tables  were  obtained  through  the  Office  of  the  Under 
Secretary  of  Defense  (Comptroller)  website  (Office  of  the  Under  Secretary  of  Defense, 
2007).  The  civilian  pay  was  comprised  of  Non-appropriated  Funds  (NAF)  pay  tables. 
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Appropriated  Funds  (AF)  pay  tables  and  General  Schedule  (GS)  pay  tables.  These  civilian 
base-pay  tables  were  obtained  from  the  Department  of  Defense  (DoD)  Civilian  Personnel 
Management  Service  Website,  and  the  fringe  rates  were  obtained  from  the  Office  of 
Management  and  Budget  (0MB)  transmittal  M-07-02  (Civilian  Personnel  Management, 
2007).  The  SCA  pay  tables  and  fringe  rates  were  obtained  from  the  DoL-  SCA  Wage 
Determinations  website  (Wage,  2007). 

Next,  all  the  base  and  composite  pay  tables  were  aggregated  by  year  and  the  pay 
increases  for  individual  positions  or  grades  were  calculated.  The  mean  increase  for  each 
position  or  grade  was  computed  to  form  an  overall  rate  of  increase  for  each  year.  This 
process  was  repeated  for  the  civilian,  SCA  and  CBA  pay  at  the  seven  different  bases  and 
the  results  were  tabulated  and  graphed."*^ 

Analysis  of  the  four  pay  types  occurred  in  three  phases.  First,  the  average  pay 
increases,  by  position,  for  each  pay  type  were  analyzed  by  year.  This  analysis  was  not 
conclusive  because  pay  types  such  as  SCA  didn’t  get  pay  raises  every  year.  Often,  the 
SCA  pay  increases  occurred  every  two  years  in  large  increments.  Thus,  the  graphs 
resulting  from  this  phase  of  analysis  were  not  static  and  did  not  reveal  any  significant  trends. 

The  next  phase  of  the  analysis  was  to  calculate  the  cumulative  wage  increases  for 
each  pay  type.  This  was  done  by  compounding  each  pay  increase  by  that  of  previous  years 
to  reveal  the  overall  growth  over  time  for  each  position.  A  challenge  was  encountered  in  this 
phase  of  the  analysis:  not  all  bases  had  CBA  wage  history  from  FY99  to  FY06.  Thus,  the 
cumulative  wages  for  these  bases  did  not  start  until  the  CBA  history  began,  which  created  a 
lag  in  the  graphs. 

The  final  phase  of  the  analysis  was  to  adjust  the  CBA  wage  increases  for  years 
without  CBA  wage  history.  To  better  represent  the  CBA  wage  increases,  the  CBAs  were 
assumed  to  follow  SCA  wage  increases  for  the  years  that  the  CBAs  were  not  in  effect.  This 
was  a  safe  assumption  because  contracted  employees  would  have  followed  SCA  wage 
rates  before  unionization.  Also,  Tyndall  AFB  did  not  have  any  SCA  or  CBA  wage  history 
available  for  FYOO  or  FY01 .  To  account  for  this,  SCA  and  CBA  wage  increases  were  each 
conservatively  assumed  to  be  3.0%  for  both  years. 

The  results  for  all  seven  bases  were  combined  and  are  shown  in  Figures  1  and  2. 
The  Bureau  of  Labor  Statistics  (BLS)  line  on  each  graph  represents  the  DoL-reported  wage 
inflation  rate,  nation-wide,  for  each  year. 


Individual  results  for  each  base  are  available  from  the  authors. 
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AETC  Base  Pay 


Figure  8.  AETC  Base-pay  Increases  by  Year 


AETC  Composite  Pay 
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Figure  9.  AETC  Composite-pay  Increases  by  Year 


The  next  step  was  to  look  at  the  cumulative  pay  increases  over  time.  To  do  this,  the 
rate  of  increase  for  every  year  is  compounded  by  the  rates  of  increase  for  the  previous 
years.  This  step  assists  in  showing  the  total  increase  over  time  for  each  category  of 
employee.  The  results  of  this  analysis  are  shown  in  Figures  3  and 


Individual  results  for  each  base  are  available  from  the  authors. 
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Figure  10.  AETC  Cumulative  Base-pay  Increases 


Figure  11.  AETC  Cumulative  Composite-pay  Increases 


One  challenge  existed  in  this  cumulative-pay  analysis;  some  bases  did  not  have  any 
CBAs  for  FYOO,  FY01  or  FY02.  To  better  represent  the  CBA  wage  increases,  the  CBAs  are 
assumed  to  follow  SCA  wage  increases  for  the  years  that  the  CBAs  were  not  in  effect.  This 
is  a  safe  assumption  because  contracted  employees  would  have  followed  SCA  wage  rates 
before  unionization.  Also,  as  mentioned  previously,  Tyndall  AFB  did  not  have  any  SCA  or 
CBA  wage  history  available  for  FYOO  or  FY01 .  To  account  for  this,  SCA  and  CBA  wage 
increases  are  each  conservatively  assumed  to  be  3.0%  (lower  than  the  BLS)  for  both  years. 
The  results  of  these  adjustments  are  shown  in  Figures  5  and  6. 
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AETC  Cumulative  Base-Pay  (C BA  Adjusted) 
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Figure  12.  AETC  Cumulative  Base-pay  Increases  (CBA  Adjusted) 


Figure  13.  AETC  Cumulative  Composite-pay  Increases  (CBA  Adjusted) 


Figures  5  and  6  show  that  CBAs  have  increased  at  a  slightly  higher  rate  than  the 
SCA  wages,  and  the  military  pay  rates  have  increased  at  a  higher  rate  than  any  of  the  other 
pay  types.  Table  1  shows  the  cumulative  base-  and  composite-pay  increases  from  FY99  to 


FY06. 
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Pay  Type 

Cumulative  Base-pay 
Increases 

Cumulative  Composite-pay 
Increases 

Military 

38.98% 

50.03% 

CBA 

34.99% 

38.87% 

Civilian 

28.21% 

31.65% 

SCA 

22.63% 

32.35% 

BLS 

24.60% 

24.60% 

Table  1. 

AETC  Cumulative  Pay  Increases  from  FY99  to  FY06 

Objective  2:  Contractor  vs.  Military  Cost  Growth 

Seven  AETC  bases  have  contracted  out  for  their  fuels  support  (Columbus,  Laughlin, 
Maxwell,  Randolph,  Sheppard,  Tyndall,  and  Vance).  All  of  these  workforces  operate  under 
collective  bargaining  agreements.  However,  two  of  these  bases  (Maxwell  and  Tyndall) 
operate  under  BOS  (Base  Operations  Support)  contracts  that  combine  many  functional 
areas  under  one  contract  line-item.  This  made  the  fuels  costs  at  these  two  bases  difficult  to 
clearly  identify.  As  a  result.  Maxwell  and  Tyndall  AFBs  were  eliminated  from  the  study.  Six 
AETC  bases  perform  the  function  in-house  with  military  and/or  civilian  personnel  (Altus, 
Goodfellow,  Keesler,  Lackland,  Little  Rock,  and  Luke). 

The  manpower  standard  for  the  fuel  function  is  not  an  accurate  predictor  of  the 
number  of  personnel  required  for  smaller  workloads  because  it  starts  with  ~27  positions  as  a 
baseline  regardless  of  the  workload.  Thus,  because  Goodfellow  AFB  handles  considerably 
less  fuel  than  the  other  AETC  bases  (which  skews  the  data),  it  was  excluded  from  the 
analysis."'^  The  remaining  pool  for  analysis  consisted  of  five  in-house  bases  and  five 
collective  bargaining  agreement  bases.  With  the  assistance  of  the  Manpower  flight  at 
Maxwell  AFB  and  the  fuels  functional  chief  at  AETC,  the  standard  manpower  costs  for  each 
of  these  ten  organizations  were  calculated  using  the  composite-pay  figures  for  military  and 
civilian  personnel.  The  results  are  in  Figure  7. 

Figure  7  indicates  that  the  Fuels  function  is  varied  at  the  eight  bases  examined. 

From  Altus,  with  an  annual  calculated  manpower  standard  cost  of  nearly  $5  million  dollars, 
to  Keesler,  with  an  annual  calculated  manpower  standard  cost  of  just  over  $2  million,  the 
workload  is  wide-ranging  at  AETC  locations. 


At  Goodfellow,  all  fuels  functions  are  performed  by  one  person,  a  GS-7. 
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Manpower  standard  cost 
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Figure  14.  Manpower  Standard  Cost 


The  fuels  function  at  ten  AETC  bases  was  analyzed  by  comparing  actual  FY06  costs 
to  their  respective  manpower  standard-derived  costs.  Five  of  the  bases  analyzed  had 
contracted  fuel  organizations,  and  the  other  five  had  in-house  fuel  organizations.  In  the  first 
step,  manpower  standards  were  calculated  using  the  Fuels  Management  Manpower 
Standard  outlined  in  AFMS  41  DA.  The  resulting  manpower  standards  were  then  converted 
to  positions  as  described  in  AFMS  41  DA,  and  the  positions  were  next  converted  to  total 
cost. 


Next,  the  actual  cost  of  each  organization  was  compared  to  the  calculated  manpower 
standard  cost.  For  contracted  organizations,  the  actual  FY06  contract  costs  were  used  in 
the  comparison.  For  in-house  organizations,  the  assigned  personnel  numbers  obtained 
from  the  AFPC  Authorized  and  Assigned  database  were  used  to  compute  the  total 
personnel  costs. 

Finally,  the  cost  ratios  of  actual  cost  and  manpower  standard-calculated  costs  were 
calculated,  revealing  that  contractor  costs  remain  economically  advantageous  to  the  Air 
Force. 


The  next  step  in  the  calculation  was  to  calculate  the  actual  cost  of  manpower  at 
those  locations  that  are  manned  by  in-house  personnel.  With  assistance  from  AETC  fuels 
personnel  and  the  Air  Force  Personnel  Center’s  Assigned/Authorized  data  source,  the 
actual  cost  for  labor  for  FY06  was  calculated  for  each  of  the  five  bases.  Again,  the 
composite  pay  rates  are  used  in  this  calculation.  The  results  are  in  Figure  8. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  -  325  - 


$7,000,000 

$6,000,000 

$5,000,000 

$4,000,000 

$3,000,000 

$2,000,000 

$1,000,000 

$0 


Figure  15.  Manpower  Standard  Costs  vs.  In-house  Costs  at  In-house  Organizations 


□  Manpower  standard  cost 


□  Authorized  Cost 


Altus  Keesler  Lackland  Little  Rock  Luke 


A  similar  comparison  was  then  made  between  the  contracted  organizations  and  their 
calculated  manpower  standard  costs.  This  comparison  is  shown  in  Figure  9.  Table  2  then 
compares  the  percentage  of  the  calculated  manpower  standard  cost  to  the  actual 
organization  cost.  This  comparison  uses  the  assigned-personnel  cost  for  in-house 
organizations,  as  this  more  accurately  captures  the  cost  to  the  Air  Force. 
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Figure  16.  Manpower  Standard  Cost  vs.  Contract  Cost  at  Contracted  Organizations 


□  Manpower  standard  cost 
■  Contract  Cost 


Columbus  Laughlin  Randolph  Sheppard  Vance 


Base 

In-house 

Contracted 

Percentage 

Altus 

X 

97.8% 

Columbus 

X 

66.2% 

Keesler 

X 

69.2% 

Lackland 

X 

127.2% 

Laughlin 

X 

46.1% 

Little  Rock 

X 

147.5% 

Luke 

X 

162.2% 

Randolph 

X 

59.4% 

Sheppard 

X 

54.6% 

Vance 

X 

76.6% 

Table  2.  Percent  of  Actual  Cost  vs.  Manpower  Standard  Cost 


The  in-house  bases  averaged  123.0%  of  the  hypothetical  manpower  standard  cost. 
As  seen  in  Table  2,  Keesler  AFB  is  the  only  in-house  organization  to  perform  the  function  at 
a  considerably  lower  cost  than  dictated  by  the  manpower  standard.  This  is  possibly  a  result 
of  the  inherent  flaw  in  the  manpower  standard  for  organizations  with  smaller  workloads.  The 
contracted  organizations  averaged  60.7%  of  the  hypothetical  manpower  standard  cost. 

Thus,  using  the  manpower  standard  cost  simply  to  contrast  how  in-house  and  contractor 
actual  costs  compared  to  the  standard,  contractor  costs  are  still  economically  advantageous 
overall  for  the  sampled  fuel  functions. 
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Findings 


Objective  1 :  Cost  Growth  of  Labor  Rates 

There  was  an  observed  trend  of  CBA  wages  growing  at  a  faster  rate  than  SCA 
wages.  Typically,  annual  CBA  base-pay  increases  were  1 .76%  higher  than  the  SCA  base- 
pay  increases.  The  analysis  showed  that  the  annual  composite-pay  increases  were  ~1% 
higher  in  CBA  positions  than  in  SCA  positions.  This  trend  is  probably  more  significant  than 
shown  considering  the  initial  jump  in  wages  that  often  occurs  when  SCA  positions  unionize. 
When  positions  unionize,  the  CBA  usually  negotiates  a  one-time  spike  in  wages  before 
leveling  off.  This  SCA-to-CBA  wage  jump  was  not  captured  by  our  analysis  because  the 
conversion  usually  involved  a  position-name  change,  preventing  direct  comparison.  The 
military  and  CBA  wage  increases  appeared  to  be  the  fastest  growing  of  the  four  pay-types 
investigated  at  the  seven  bases. 

Objective  2:  Contractor  vs.  Military  Cost  Growth 

There  was  a  trend  that  contractors  performed  fuel  functions  at  a  lower  cost  than  in- 
house  in  comparison  to  their  calculated  manpower  standard  costs.  For  contracted  services, 
the  actual  cost  of  contractors  averaged  -40%  less  than  the  cost  derived  from  the  manpower 
standard.  For  in-house  services,  the  actual  in-house  cost  averaged  over  20%  higher  than 
the  cost  derived  from  the  manpower  standard.  Thus,  in  these  cases,  the  A-76  actions  still 
appear  to  be  economically  advantageous  to  the  government. 


Limitations 

1 .  Manpower  data  is  difficult  to  obtain  outside  of  a  limited  timeframe.  This  limits  the  ability 
to  develop  an  in-house  organization  comparable  to  a  contracted  organization  without 
significant  investment  of  time  and  expertise. 

2.  Contract  data  is  limited  in  its  timeframe.  Contracting  files  are  maintained  in  accordance 
with  the  Federal  Acquisition  Regulation.  The  timeframe  of  the  study  is  limited  to  the 
years  that  the  data  is  available. 

3.  There  is  a  limited  sample  size  available  for  the  study.  It  only  includes  those  service 
contracts  within  AETC  that  have  CBAs.  The  size  was  further  constrained  to  those 
locations  where  multiple  contracts  exist.  This  limitation  and  constraint  restricts  the 
general  application  of  the  results  of  the  study. 

4.  Changes  in  position  names/job  titles  when  contractor  positions  unionize  prevent  a  direct 
comparison  of  SCA  and  CBA  wage  growth. 

5.  All  military  and  civilian  pay  grades  were  analyzed,  but  not  all  pay  grades  would  typically 
be  converted  to  service  contract  positions. 

Conclusion 

This  study  takes  important  preliminary  steps  toward  determining  whether  an 

additional  A-76  competitive  sourcing  process  factor  to  account  for  future  contract  cost 

growth  should  be  developed.  We  found  the  research  highly  challenging  due  to  the  lack  of 
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historical  documentation  in  all  areas.  The  documents  recording  the  manpower  assessment 
at  the  heart  of  the  A-76  process  is  almost  never  available.  The  assumptions  used  by 
evaluators  relative  to  future  labor  cost  growth  at  the  time  of  the  A-76  decision  are  difficult  to 
acquire,  if  not  impossible.  As  such,  it  is  difficult  if  not  impossible  (perhaps  by  design)  to 
conduct  an  assessment  of  how  closely  the  assumptions  at  the  time  of  the  A-76  track  the 
actual  post-decision  labor  increases.  As  such,  it  would  be  difficult  to  develop  a  factor  to 
adjust  the  assumptions  used  by  evaluators  if  we  cannot  determine  what  the  initial 
assumptions  were. 

A  common  perception  in  the  DoD  is  that  CBA  cost  growth  in  the  out-years  makes 
outsourcing  less  attractive  financially.  This  study  finds  that  while  CBA  cost  growth  is 
substantial,  in  our  sample  functional  area,  CBA  cost  growth  lagged  behind  the  cost  growth 
of  military  labor.  One  explanation  may  be  that  while  commanders  are  forced  to  deal  with  the 
realities  of  increasing  contract  costs  at  the  MAJCOM  and  local  level,  they  may  be  less  aware 
of  the  challenges  of  dealing  with  the  even  more  substantial  increases  in  military  personnel 
costs,  which  are  often  dealt  with  at  the  Air  Staff  level. 

A  final  finding  of  note  is  that  government  service  civilian  cost  growth  lagged  behind 
both  military  and  contract  labor  growth,  indicating  that  over  time,  civilian  labor  may  prove  to 
be  the  most  stable  of  the  three  labor  types  considered.  Our  findings  do  indicate  that  future 
cost  growth  is  not  stable;  however,  it  appears  to  be  instable  in  both  military  and  contractor 
labor  pools.  As  such,  should  a  factor  be  pursued  to  provide  better  insight  into  the  future  cost 
of  contractor  labor,  one  should  also  be  pursued  to  provide  better  insight  into  military  labor 
cost. 
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Executive  Summary 

The  Office  of  Management  and  Budget  (0MB)  Circular  A-76,  Performance  of 
Commercial  Activities,  has  been  around  for  over  40  years.  A-76  was  commonly  used  to 
refer  to  the  process  of  outsourcing  non-inherently  governmental  jobs  in  the  1990’s.  The  A- 
76  circular  was  substantially  revised  to  simplify  and  standardize  implementation  guidance 
and  was  released  in  its  current  form  on  May  29,  2003.  The  0MB  now  uses  A-76  to  help 
implement  the  competitive  sourcing  initiative  on  President  Bush’s  Management  Agenda. 

The  stated  policy  objective  in  the  revised  circular  is,  “To  ensure  that  the  American  people 
receive  maximum  value  for  their  tax  dollars,  commercial  activities  should  be  subject  to  the 
forces  of  competition”  (0MB,  2003,  May  29).  The  purpose  of  this  study  is  to  determine  if  A- 
76  competitions  have  resulted  in  saving  the  taxpayers  money.  More  specifically,  has  A-76 
saved  the  Department  of  Defense  (DoD)  money  over  the  long  term? 

The  thinking  goes  that  by  subjecting  the  non-inherently  governmental  jobs  being 
performed  by  government  employees  to  the  “forces  of  competition,”  the  American  taxpayer 
should  benefit  in  the  form  of  better  service  at  lower  cost.  As  stated  in  the  Federal 
Acquisition  Regulation  (FAR),  “Commercial  activities  should  be  subjected  to  the  forces  of 
competition”  (Part  7.302).  In  this  study,  I  will  be  primarily  concerned  with  the  lower  cost 
expectation  and  some  of  the  broader  policy  implications  of  this  strategy. 

So  the  question  is:  have  A-76  competitions  saved  the  DoD  money?  A  General 
Accounting  Office  (GAO)  report  in  2000  on  A-76  studies  in  the  DoD  concluded  that  the 
Department  had  saved  $290  million  dollars  on  286  studies  conducted  from  1995  to  1999. 
However,  the  study  goes  on  to  say  that  there  were  some  problems  in  the  way  the 
Government  calculated  baseline  costs,  which  were  often  based  on  authorized  positions 
rather  than  on  actual  positions,  and  that  the  cost-savings  estimates  did  not  include  the  costs 
of  conducting  the  studies  or  the  cost  of  implementing  and  managing  them  (GAO,  2000).  In 
other  words,  true  cost  savings  must  consider  the  costs  of  conducting  the  studies  and  the 
costs  of  administering  the  contracts  or  otherwise  monitoring  performance.  It  was  also, 
“noted  that  the  level  of  savings  will  be  difficult  to  track  in  the  long  term  because  workload 
requirements  change,  affecting  program  costs  and  the  baseline  from  which  savings  are 
calculated”  (2000). 
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A  more  recent  study  conducted  by  the  IBM  Center  for  The  Business  of  Government 
found  that  competition  results  in  significant  savings.  This  study  examined  competitions 
conducted  from  1994  though  the  first  quarter  of  2004  and  used  the  DoD’s  own  data- 
collection  system  to  conclude  that  there  was,  “an  average  estimated  savings  of  44  percent 
of  baseline  costs,  for  a  total  savings  of  $1 1 .2  billion”  (Gansler  &  Lucyshyn,  2004,  October,  p. 
28,  32).  The  study  also  found  that  although  there  were  relatively  few  government  civilian 
employees  involuntarily  separated,  most  of  the  savings  associated  with  the  competitions 
came  as  a  result  of  reducing  the  number  of  positions  by  24,852  (2004,  October).  The 
number  of  positions  was  reduced  regardless  of  who  won  the  competition — the  government 
or  a  contractor.  The  savings  estimate  was  also  based  on  estimated  cost  and  did  not 
consider  the  cost  of  conducting  the  study  or  administering  the  contracts. 

In  another  report  (GAO-04-367),  the  GAO  indicates  that  although  the  DoD  has 
achieved  savings  though  competitive  sourcing,  it  is  difficult  to  estimate  precisely  the  amount 
of  these  savings  (2004).  Further,  the  Office  says  significant  challenges  face  the  agencies 
implementing  competitive  sourcing,  including:  difficulty  in  identifying  non-inherently 
governmental  positions  for  competition,  a  constantly  changing  environment,  insufficient  staff 
to  plan  and  carryout  the  competitions  and  a  lack  of  funding  to  implement  and  administer  the 
program  (2004). 

In  a  widely  publicized  White  Paper,  the  Office  of  Management  and  Budget  (0MB) 
looked  at  217  competitive  sourcing  studies  conducted  in  FY  2004  and  estimated  these 
studies  would  “generate  a  net  savings,”  or  cost  avoidance  of  $1 .4  billion  over  three  to  five 
years  (2005,  January).  They  claim  that  the  data  suggests  the  savings  are  primarily  because 
of  larger  competitions  and  more  frequent  use  of  standard  competitions  that  require  in-house 
teams  to  come  up  with  a  most  efficient  organization.  Put  another  way,  “competition  drives 
bid  prices  down  and  efficiencies  up”  (2005,  January).  It  remains  to  be  seen  whether  or  not 
this  will  remain  true  over  the  life  of  the  contracts. 

An  interesting  study  by  David  Galley  (2002),  an  adjunct  faculty  member  at  George 
Washington  University,  examined  the  impact  of  public-private  competitions  on  the  costs  of 
providing  maintenance  and  repair  services  for  buildings  on  104  Army  installations  from  1989 
to  1999.  He  says  that  the  study,  “shows  conclusively  that  the  impact  of  those  A-76 
competitions  on  costs  depended  on  the  category  of  the  winner”  (pp.  3-18).  If  the  private 
sector  won,  there  was  significant  cost  savings;  but  if  the  work  was  kept  in-house,  there  was 
not  (2002).  One  offered  explanation  for  this  was  that  the  Army  did  not  monitor  the  in-house 
work  force’s  performance  like  it  did  when  a  contractor  was  involved. 

A  study  at  the  US  Air  Force  Academy  examined  five  different  studies  conducted 
during  FY  2001  to  2003  which  resulted  in  significantly  reducing  the  number  of  employees 
performing  competitive  sourcing  functions.  However,  a  survey  of  employees  and  managers 
familiar  with  the  competitive  sourcing  programs  indicated  that  80%  felt  the  services  were 
worse  than  before  the  studies  had  been  done  and  only  about  half  thought  the  program 
saved  money  for  the  Air  Force  (Green,  Heppard  &  Forrester,  2004,  pp.  4-1 1 ).  In  one 
conclusion,  this  study  also  indicated  that,  “Estimates  used  to  compute  savings  omit  many 
costs,  e.g.,  study  costs,  retraining  costs,  loss  of  productivity,  severances  packages,  etc.” 
(2004). 

In  a  briefing  paper  entitled  Show  Me  The  Money,  Max  Sawicky  of  the  Economic 
Policy  Institute  investigated  the  evidence  presented  by  the  Office  of  Management  and 
Budget  (0MB)  in  support  of  the  Revised  A-76  Circular  that  purported  to  show  that  A-76 
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competitions  saved  money.  One  of  the  studies  cited  in  this  paper  was  conducted  at  the 
Center  for  Naval  Analysis  (CNA)  and  involved  16  competitions  for  2,800  jobs.  His  findings 
indicated  that  in  every  case,  the  “observed  cost”  (or  the  actual  cost  of  performance) 
exceeded  the  bid  price.  The  “effective  cost”  (the  actual  cost  taking  into  consideration  the 
change  in  the  scope  of  work)  exceeded  the  bid  price  in  nine  out  of  fourteen  cases  (Sawicky, 
2003).  An  interesting  observation  in  this  study  was  that  even  if  there  are  apparent  savings, 
they  are  not  returned  to  the  taxpayer.  In  effect,  competitions  are  really  a  vehicle  for 
expanded  government  because,  “agency  savings  do  not  translate  into  budget  savings” 
(2003,  p.  8,  12). 

In  light  of  these  and  other  studies  on  outsourcing  in  the  DoD,  it  is  interesting  that  a 
recent  article  in  the  Federal  Times  asserts  that  according  to  a  Pentagon  official,  “it’s  unclear 
if  the  military  always  saves  money  when  it  contracts  with  private  companies  to  perform 
support  services.”  Claude  Bolton,  assistant  secretary  of  the  Army  for  acquisition,  logistics 
and  technology,  is  quoted  as  saying  that,  “While  it  may  be  clear  that  a  particular  contract  will 
save  millions  of  dollars  right  away,  there  is  no  simple  way  to  determine  if  that  remains  true 
four  or  five  years  down  the  road”  (as  cited  in  Lubold,  2006,  p.  13).  In  other  words,  estimated 
savings  at  the  beginning  may  not  translate  into  actual  savings  in  the  end. 

“Ah,  there’s  the  rub.”  Federal  employees  unions  and  others  have  been  making  this 
case  for  years.  Testifying  before  Congress,  the  president  of  the  American  Federation  of 
Government  Employees,  Bobby  Harnage  (2001,  March  15),  indicated  that  even  as  the 
number  of  federal  government  employees  in  the  DoD  steadily  decreased  by  over  280,000 
from  1992  to  1999,  the  cost  of  service  contracts  increased  from  $39.9  billion  to  $51 .8  billion 
(p.  1).  In  the  DoD,  service  contracts  are  now  the  largest  segment  of  DoD  procurements  and 
make  up  over  50%  of  all  contract  dollars  awarded  (Lubold,  2006,  April  24,  p.  13).  In  fact, 
according  to  a  recent  congressional  report,  spending  on  federal  contracts  has  been  the 
fastest  growing  part  of  the  discretionary  budget  for  the  last  five  years,  and  service  contracts 
are  leading  the  way  (US  House,  2006).  As  anyone  familiar  with  the  federal  budget  can  tell 
you,  the  budget  has  not  gone  down  regardless  of  how  many  jobs  are  contracted  out  or  how 
many  activities  are  turned  over  to  contractors.  In  the  DoD,  budgets  have  increased  almost 
every  year  since  1994,  and  they  are  not  expected  to  go  down  anytime  soon  (0MB,  2005). 

So,  does  outsourcing  or  competitive  sourcing  really  save  the  taxpayer  money?  It 
may  save  money  on  some  contracts,  but  there  is  a  question  about  whether  the  government 
customer  is  getting  the  same  or  better  service.  Also,  projected  savings  and  actual  savings 
are  two  entirely  different  things.  The  comptroller  general  of  the  United  States,  David  Walker, 
was  quoted  recently  as  saying,  “they  (contractors)  often  fail  to  deliver  the  promised 
efficiency  and  savings.  Private  companies  cannot  be  expected  to  look  out  for  taxpayers’ 
interests”  (as  cited  in  Shane,  2007,  February  4).  Generally,  it  does  not  appear  A-76  is 
saving  the  taxpayer  money  overall.  As  pointed  out  above,  the  federal  budget  is  not 
decreasing,  in  spite  of  the  hundreds  of  thousands  of  jobs  that  have  been  contracted  out. 

Both  Congress  and  the  Executive  branch  like  to  tell  voters  they  are  cutting  the  bureaucracy, 
but  in  reality  they  are  merely  transferring  government  functions  to  private  contractors.  This 
growing  “shadow  workforce”  has  been  identified  by  many  writers,  but  Paul  Light  of  the 
Brookings  Institute  has  been  a  leader  in  this  area;  he  has  an  excellent  book  on  the  subject 
entitled  The  True  Size  of  Government,  in  which  he  concludes,  “the  true  size  of  the 
government  is  much  larger  than  the  federal  employee  headcount  suggests”  (1999,  p.  44). 

There  is  an  old  saying  in  Washington  that,  “there  are  two  kinds  of  contractors,  those 
with  government  contracts  and  those  that  want  government  contracts.”  The  fact  that  the 
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federal  government  is  the  largest  purchaser  of  goods  and  services  in  the  world  makes  it  a 
very  attractive  customer.  The  implication  that  by  subjecting  non-inherently  governmental 
jobs  to  the  forces  of  competition  somehow  captures  market  efficiencies  is  on  its  face 
plausible;  but  on  closer  examination,  this  implication  may  be  more  apparent  than  real.  First, 
as  stated  by  Professor  Dan  Guttman  of  Johns  Hopkins  University  in  a  recent  article, 
although,  “we  associate  the  utility  of  contractors  with  the  notion  that  the  private  sector  brings 
market  forces  to  bear  on  government  activities,  this  is  only  true  where  a  commercial  market 
exists  for  the  government  purchases”  (2004,  p.  24).  He  goes  on  to  say,  “where  government 
is  the  primary  or  predominant  purchaser  of  services  or  goods  the  picture  is  less  clear” 

(2004).  As  has  been  demonstrated  by  Peck  and  Sherer  (1962)  In  their  classic  analysis  of 
weapon  system  procurement,  markets  are  difficult  to  duplicate  in  a  governmental  setting 
primarily  because  the  government  entity  really  doesn’t  have  any  market  competition  for 
many  of  the  products  they  buy.  As  they  point  out,  the  government  is  often  the  only  buyer  (p. 
60).  This  rational  can  also  be  applied  to  services  the  government  purchases.  For  instance, 
typically  the  work  (services)  is  to  be  performed  on  a  government  installation;  the  work  will  be 
performed  during  certain  hours,  and  the  work  is  controlled  by  a  performance  work  statement 
or  statement  of  work  with  some  type  of  government  monitoring.  The  services  may  even  be 
performed  in  a  combat  environment.  This  is  not  exactly  “market  conditions.” 

Secondly,  as  has  been  asked  by  others  and  included  in  Dan  Guttman’s  article,  “In 
what  respect  is  a  contractor  that  earns  nearly  1 00%  of  its  income  from  doing  government 
work  engaged  in  a  commercial  activity?”  (2004,  p.  29,  quoting  from  the  “Bell  Report”).  Third, 
if  the  government  can  manage  its  organization  better  by  creating  a  “most  efficient 
organization,”  why  isn’t  it  doing  so  already? — because,  as  has  been  shown  (See  IBMCBG 
study  cited  above),  most  of  the  alleged  savings  comes  from  reducing  the  number  of 
employees.  Fourth,  the  whole  debate  about  what  is  inherently  governmental  varies  from 
time  to  time  and  place  to  place.  There  is  conflicting  guidance  from  agencies  on  what  is 
inherently  governmental,  and  some  of  it  even  seems  to  be  in  conflict  with  the  Federal 
Acquisition  Regulations.  And  finally,  social  and  economic  policies  like  equal  employment 
opportunity,  veteran’s  preference,  providing  jobs  for  the  handicapped  and  disabled,  health 
insurance  and  retirement  benefits  for  employees  are  now  discounted  in  the  name  of 
competition.  Is  this  really  what  “competitive  sourcing”  is  supposed  to  be  about? 

Competitive  sourcing  (or,  as  it  used  to  be  called,  “commercial  activities”)  is  about 
private-sector  contractors  doing  commercial  work.  This  should  not  be  that  hard  to  define. 
Like  Lt.  General  Donald  Hoffman,  military  deputy  in  the  office  of  the  assistant  secretary  of 
the  Air  Force  said,  “We  don’t  need  Air  Force  plumbers  and  Air  Force  electricians  when  right 
outside  the  gate  there  is  a  contractor  to  do  that  stuff”  (as  cited  in  Lubold,  2006,  p.  13).  I 
think  he  is  right.  Those  are  excellent  examples  of  commercial  activities.  But  simply 
because  some  contractors  want  more  government  work,  and  some  government  officials 
want  to  give  it  to  them  (for  a  variety  of  reasons),  that  still  doesn’t  make  almost  everything  the 
government  does  a  commercial  activity.  It  is  high  time  that  the  public  and  their  elected 
representatives  recognize  that  reducing  the  size  of  government  is  about  more  than 
eliminating  federal  employees  before  it’s  too  late.  In  fact,  a  good  first  step  might  be  for 
congress  to  take  the  recommendation  of  their  own  analysts  and,  “further  prescribe  that 
certain  government  activities  are  to  be  considered  inherently  governmental”  (Grasso,  2003, 
October  20,  p.  23).  This  would  remove  the  arbitrary  discretion  of  understaffed  agencies 
looking  for  a  way  to  outsource  their  work,  and  perhaps  save  us  from  ourselves,  while  making 
sure  all  the  agencies  use  the  same  criteria  when  determining  what  jobs  are  to  be 
outsourced. 
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Although  it  may  be  unpopular  to  say  that  the  government  has  a  higher  calling  than 
the  purely  monetary  incentive  of  business  to  make  a  profit,  it  is  nevertheiess  true.  The 
government  typically  has  a  mission  to  accomplish,  and  it  may  not  be  financiaily  feasible  from 
a  business  point  of  view.  When  this  is  the  case,  why  should  taxpayers  be  expected  to  not 
only  pay  for  the  service  but  to  aiso  pay  a  profit?  This  is  especiaiiy  true  when  the  services 
being  performed  are  not  realiy  commerciai  services.  For  competitive  sourcing  to  reaiiy  work, 
the  first  step  must  be,  as  Professor  Steven  Schooner  has  indicated,  “to  determine  which 
functions  are  inherently  governmental  and  which  are  not”  (2004,  p.  295).  It  should  not  be 
that  difficult  to  identify  a  commercial  service  or  activity.  Then,  “the  government  shouid  begin 
outsourcing  those  services  that  are  most  readiiy  avaiiabie  in  the  private  sector”  (2004,  p. 
296).  This  is  the  oniy  way  the  government  can  truly  benefit  from  “market  forces.”  To  try  to 
compete  jobs  that  are  not  typically  performed  in  the  private  sector  is  not  in  the  public  interest 
and  wiil  not  save  the  taxpayer  money  in  the  long  run. 

Beyond  the  cost  savings  or  cost  avoidance  that  may  or  may  not  be  associated  with 
competitive  sourcing,  there  is  the  question  of  what  is  in  the  best  interest  of  the  government 
and  the  pubiic.  On  this  point,  an  outsourcing  survey  conducted  by  Deloitte  Consulting 
Corporation  of  25  world  class  organizations  found  that,  “outsourcing  is  an  extraordinariiy 
compiex  process  and  the  anticipated  benefits  often  faii  to  materiaiize”  (2005,  Aprii,  pp.  3-4). 
Further,  they  say,  “In  the  long  run,  organizations  that  continue  to  outsource  wiil  experience  a 
loss  of  bargaining  power  to  vendors  as  the  suppiy  side  consoiidates”  (2005,  April).  Both  of 
these  phenomena  have  occurred  in  the  Defense  industry  in  over  the  past  20  years  and,  “it  is 
unlikely  that  the  defense  industry  wiii  ever  approximate  a  competitive  market”  (Driessnack  & 
King,  2004,  January-Aprii).  As  the  supply  side  continues  to  consoiidate  domesticaliy  and 
internationally,  is  it  reaiiy  in  our  national  interest  to  contract  out  more  and  more  government 
activities?  Perhaps  the  debate  we  shouid  be  having  is  what  is  in  our  national  best  interest 
from  the  taxpayer’s  point  of  view — because  as  we  have  seen,  the  cost  of  government  is  not 
going  down. 
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Panel  10  -  Evolving  Technologies  in  DoD  Acquisition 


Wednesday, 
May  16,  2007 


3:30  p.m.-  Chair: 
5:00  p.m. 


Lorna  B.  Estep,  Deputy  Director  for  Supply,  Directorate  of  Logistics  and 
Sustainment,  Headquarters  Air  Force  Materiel  Command 

Discussant: 

Michaei  A.  Schwind,  Vice  President,  Federal  Sector,  UGS  Corporation 

Papers: 

Optimal  Inventory  Policy  for  Two-echelon  Remanufacturing 
Geraldo  Ferrer,  Naval  Postgraduate  School 
B2B  Models  for  DoD  Acquisition 

Magdi  N.  Kamel,  Naval  Postgraduate  School 


Chair:  Lorna  B.  Estep,  Deputy  Director  for  Supply,  Directorate  of  Logistics  and  Sustainment, 
Headquarters  Air  Force  Materiel  Command,  is  responsible  for  the  Materiel  Support  Division  of  the 
Supply  Management  Activity  Group,  a  stock  fund  with  annual  sales  of  $7  billion.  She  directs  a  wide 
range  of  logistics  services  in  support  of  Air  Force  managed  spare  parts,  to  include  transformation 
programs,  requirements  determination,  budgeting,  acquisition,  provisioning,  cataloging,  distribution 
and  data  management  policy.  She  also  provides  supply  chain  management  policy,  guidance  and 
direction  in  support  of  headquarters,  air  logistics  centers,  and  U.S.  Air  Force  worldwide  customers. 

Ms.  Estep  started  her  career  as  a  Navy  logistics  management  intern.  She  has  directed  the  Joint 
Center  for  Flexible  Computer  Integrated  Manufacturing,  was  the  first  program  manager  for  Rapid 
Acquisition  of  Manufactured  Parts,  and  has  served  as  Technical  Director  of  Information  Technology 
Initiatives  at  the  Naval  Supply  Systems  Command.  In  these  positions  she  has  developed  logistics 
programs  for  the  Department  of  Defense,  implemented  one  of  the  first  integrated  and  agile  data- 
driven  manufacturing  systems,  and  directed  the  development  of  complex  technical  data  systems  for 
the  Navy. 

As  the  Director  of  Joint  Logistics  Systems  Center,  Ms.  Estep  had  the  duties  of  a  commanding  officer 
for  a  major  subordinate  command.  In  addition,  she  acted  as  the  Logistics  Community  Manager,  an 
emerging  organization  to  coordinate  and  implement  the  revised  Defense  Department  logistics 
strategy  for  achieving  Joint  Vision  2010  through  modern  information  techniques  and  processes.  She 
has  also  served  as  Chief  Information  Officer  for  the  Naval  Sea  Systems  Command  in  Arlington,  Va., 
and  Executive  Director  of  Headquarters  Materiel  Systems  Group  at  Wright-Patterson  AFB.  Prior  to 
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Abstract 

We  present  a  two-echelon  remanufacturing  facility  subject  to  constant  demand, 
in  which  the  disassembly  process  and  the  repair  process  observe  stochastic  yield.  We 
develop  an  intuitive  scheduling  policy  and  perform  a  robustness  test. 

Keyword:  inventory  management,  multi-echelon,  remanufacturing,  product 
recovery,  stochastic  process  yield,  financial  holding  cost,  physical  holding  cost 

Introduction 

Yano  and  Lee  (1995)  revised  several  lot-sizing  models  in  which  production  yield  is 
random.  A  large  number  of  those  models  were  inspired  by  the  difficulties  faced  in  the 
production  of  electronic  components,  where  the  production  yield  in  some  stages  may  be 
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very  low.  A  similar  situation  occurs  in  remanufacturing  sites.  Cores  entering  the 
remanufacturing  shop  enter  a  pre-selection  stage  in  which  some  disassembly  takes  place. 
The  disassembly  modules  are  stocked  close  to  the  renovation  area,  where  they  are  repaired 
and  made  ready  to  reuse.  One  particularity  of  the  remanufacturing  shop  is  the  different 
ways  that  the  inventory  held  in  stock  affects  the  operating  cost,  whether  it  is  before  or  after 
the  final  production  stage.  Most  of  the  holding  cost  in  the  upstream  operation  refers  to  the 
physical  handling  of  a  large  number  of  assemblies  that  occupy  a  significant  amount  of 
space,  but  might  not  survive  the  remanufacturing  process.  Meanwhile,  most  of  the  holding 
cost  in  the  downstream  operation  refers  to  the  opportunity  cost  of  the  resources  committed 
to  adding  value  to  the  sub-assembly  renovation. 

The  remanufacturing  shop  that  we  described  has  not  been  modeled  yet.  The  paper 
we  propose  contributes  in  this  literature  by  providing  a  simple  policy  with  two  control 
variables:  the  lot  size  in  the  upstream  operation,  and  the  echelon  multiple  used  to  identify 
the  lot  size  in  the  renovation  station.  Moreover,  it  identifies  the  conditions  under  which  the 
remanufacturing  shop  will  not  hold  inventory  between  the  two  processes,  thus  renovating  all 
cores  immediately  after  disassembly. 

We  assume  that  demand  is  constant,  and  the  lead  time  of  both  processes  is  zero. 
We  develop  the  optimal  nested  policy  and  perform  numerical  tests. 


Stochastic  Process  Yield,  Deterministic  Demand 

Consider  a  remanufacturing  shop  where  the  stock  of  cores  is  unlimited  and  freely 
available  for  recovery.  The  recovery  process  generates  remanufactured  units  of  the  widget 
corresponding  to  these  cores.  This  demand  for  the  remanufactured  widget  is  fairly  stable: 
initially,  we  consider  a  constant  demand  of  D  remanufactured  goods  per  unit  time.  The 
recovery  procedure  includes  two  stages:  a  disassembly  process  and  a  renovation  process. 
Both  operations  are  costly,  require  some  setup  and  are  subject  to  a  stochastic  output  yield. 
The  manager  has  to  decide  the  operating  policy  that  determines  the  frequency  of  the  two 
operations  (disassembly  and  renovation)  and  the  size  of  the  respective  lots,  such  that 
demand  is  always  satisfied  at  the  lowest  operating  cost.  Figure  17  illustrates  this  scenario  in 
a  tire  retreading  facility. 


Figure  17.  Material  Fiow  in  a  Tire  Retreading  Facility 

One  of  the  practical  problems  faced  by  remanufacturing  shops  is  the  constraint  in 
storage  space.  A  large  number  of  used  cores  arrive  at  the  facility  to  be  processed,  but  only 
some  of  them  become  re-usable  goods.  Hence,  physical  handling  may  represent  a 
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significant  fraction  of  the  holding  cost,  especially  in  the  earlier  stages  of  the  operation.  The 
typical  representation  of  echelon  stocking,  with  nested  saw-tooth  patterns,  represents  the 
value  added  in  each  stage.  However,  this  does  not  completely  reflect  the  importance  of 
physical  handling  in  remanufacturing.  Hence,  it  is  useful  to  identify  separately  which  stage 
is  burdened  by  the  financial  and  the  physical  inventory.  Figure  18  reflects  the  two-process 
environment,  where  both  physical  and  financial  stocks  are  present  in  a  situation  in  which 
there  are  3  renovation  cycles  per  disassembly  event. 

If  the  remanufacturing  operation  pays  for  the  cores  received  at  the  time  of  delivery, 
the  financial  holding  cost  lasts  until  the  recovered  good  is  finally  delivered  to  the  customer. 
That  is  reflected  in  the  downward  slope  of  the  financial  inventory  level  in  both  processes. 
However,  the  physical  inventory  follows  a  staircase  shape  in  the  first  process,  and  a  saw¬ 
tooth  shape  in  the  lower  process.  That  behavior  is  the  same  as  most  other  multi-echelon 
systems.  However,  the  remanufacturing  operation  is  better  represented  if  the  two  holding 
costs  are  treated  separately. 

In  Figure  18,  the  first  station  disassembles  Q  machines,  subject  to  a  certain  yield,  p^. 
We  propose  a  nested  policy  such  that  the  output  of  the  upstream  station  is  split  into  n  equal 
lots  to  be  processed  in  the  downstream  operation.  Table  3  shows  the  notation  used  in  the 
optimization  of  this  policy. 


DISASSEMBLY  ECHELON 

RENOVATION  (REPAIR)  ECHELON 

kd 

setup  cost  of  disassembly 

kr 

setup  cost  of  renovation 

hf,d 

financial  holding  cost  of 
disassembled  items 

hf,r 

financial  holding  cost  of  renovated 
items 

hph,d 

physical  holding  cost  of 
disassembled  items 

hph,r 

physical  holding  cost  of  renovated 
items 

Pd 

yield  of  the  disassembly 
operation 

Pr 

yield  of  the  renovation  operation 

Q 

core  disassembly  lot-size 

n 

number  of  renovation  cycles  per 
disassembly  event 

Table  3.  Notation 


Costs  incurred  in  the  renovation  (downstream)  process 

Considering  the  yield  in  the  disassembly  operation,  Pd  Q  ready-to-recover  items  are 
available  for  renovation  in  the  second  step.  We  choose  equal  lot  sizes  of  Pd  Q  /  n  cores  in 
each  of  the  next  n  cycles  in  the  renovation  process.  If  the  yield  realization  in  the  first 
renovation  cycle  is  Pr,  we  have  that  PrPdQ/n  items  are  produced  in  the  first  cycle,  which 
are  gradually  consumed.  Moreover,  the  renovation  cycle  lasts  PrPdQ/n  D  time  units. 
Hence,  the  holding  costs  incurred  in  the  renovation  cycle  are  given  by  the  expressions: 

Equation  1.  Financial  holding  cost  during  renovation  cycle:  P^PdPrQ 

2n  nD 

Equation  2.  Physical  holding  cost  during  renovation  cycle:  PdPrQ  PdP^ 
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There  are  different  reasons  driving  the  yield  in  each  process.  Generally,  the  yield  in 
the  renovation  process  is  due  to  process  failures,  while  the  yield  in  the  disassembly  process 
depends  on  the  quality  of  the  incoming  material,  the  used  cores.  Hence,  we  may  assume 
that  the  two  yield  distributions  are  not  correlated.  The  setup  cost  per  renovation  cycle  equals 
kr.  Therefore,  the  expected  value  of  the  renovation  cost  per  time  unit  can  be  expressed  as: 


k  tiD 

Equation  3.  £■  [renovation  cost/time]  =  E 


1 

1 

E 

-Pd- 

-Pr- 

+  ^  (hf,rE  [Pd  ]  +  hpHd-E  [Pd  [/?,]) 


Figure  18.  Financial  and  Physical  Level  in  a  Two-process  System 


Costs  incurred  in  the  disassembiy  (upstream)  process 


Prior  to  the  renovation  process,  the  used  goods  inventory  is  processed  and  pre¬ 
selected  during  the  disassembly  process.  The  duration  of  the  disassembly  process  depends 
both  on  the  yield  of  this  operation,  as  well  as  on  the  yield  of  each  subordinate  renovation 
cycle,  Pr,i{i=  1,  ...  n).  Hence, 

Equation  4.  Disassembly  cycle  length:  Pd^'yPrj_ 

D  n 

Separating  the  financial  and  the  physical  holding  cost,  we  obtain  the  expressions: 


Equation  5.  Financial  holding  cost  during  disassembly  cycle: 


h 


f,d 


QPdQYEiA. 

2  D  n 


Equation  6.  Physical  holding  cost  during  disassembly  cycle: 


2n  D  n 
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The  setup  cost  per  disassembly  cycle  equals  ka.  Considering  that  the  yield 
distributions  are  not  correlated,  and  that  the  expected  duration  of  the  disassembly  cycle  is  n 
times  longer  than  the  expected  duration  of  the  renovation  cycle,  the  expected  disassembly 
cost  per  time  units  equals: 


Equation  7.  £■  [disassembly  cost/time  unit] 


KDe 

1 

E 

1 

q( 
+  — 

Q 

-Pd- 

-Pr- 

2K 

h 


f,d 


n-'C 

n  J 


Choice  of  optimal  lot-size  at  the  disassembly  process 

Equations  3  and  7  provide  the  closed-form  expressions  for  the  relevant  inventory 
costs  at  each  process  as  a  function  of  the  lot-size  of  the  disassembly  process  (Q)  and  the 
number  of  renovation  cycles  per  disassembly  cycle  (n).  Hence,  we  can  define  K(n)  and  H(n) 
as  follows: 

Equation  8.  K(np(k,  +  <)£ [!//), ]£[!/;,,] 

Equation  9.  H(n)=  h„  _  i)+  /,  +  h^  ,E[p,]) 

n  ^  ^ 

In  addition,  we  may  write  in  compact  form  the  expected  operating  cost  per  unit  time  as: 

Equation  10.  E  \C{Q,n)\  = 

Obviously,  the  expression  is  convex  in  Q.  For  a  given  value  of  n  >  0,  the  optimal  lot- 

size  is: 


2DK{n 

«(”) 

and  the  respective  minimum  cost  is: 

Equation  12.  C*{n)  =  ^j2DK{n)H{n) 

Now,  we  have  to  identify  the  integer  value  of  n  that  minimizes  this  cost  expression.  It 
is  simple  to  show  that  such  minimization  is  equivalent  to  minimizing  X(n)  given  by  the 
expression: 

Equation  13.  X(n)  =  ^^^(hf^  -  , 


Equation  11.  Q*{n)  = 


The  value  e  R  that  satisfies  the  first-order  condition  in  the  minimization  of  the 
X(n)  expression  is: 


Equation  14.  = 


1 


^  [Pd  ](^  f,r  -  hph,d  +  hph,rE  b,])  k. 


^f,d  ^ph,d^\_Pd^ 


The  value  above  is  generally  not  integer.  If  <  1,  the  minimizing  value  is  n*  =  1 . 
Otherwise,  we  examine  two  approximations  of  rireai-  Define  riio  and  n/,„  integer  numbers  such 
that  =  max{«  e  Integer  Numbers  \n  <  }  and  + 1 . 
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Clearly,  ^  «*  =  minimizes  the  cost  function.  Otherwise,  n*  =  nm  is 

the  cost  minimizer.  Now,  we  can  identify  the  lot-size  at  the  disassembly  process  that 
minimizes  the  operating  cost  in  the  remanufacturing  site.  It  suffices  to  substitute  n*  in  the 
expression  for  K(n)  and  H(n)  and,  subsequently,  substitute  them  in  the  expression  for  Q*(n) 
to  solve  the  cost  minimization  problem. 

Discussion 

Equations  11  and  14,  combined  with  the  integrality  constraint,  identify  the  decision 
variables  that  optimize  the  nested  policy  suggested  for  this  problem.  It  gives  proper  weight 
to  the  financial  and  physical  holding  costs  faced  by  the  remanufacturing  firm.  Equation  14 
shows  that  the  number  of  renovation  cycles  is  proportional  to  the  ratio  between  the  setup 
costs  of  both  processes.  The  same  result  is  observed  with  the  basic  two-echelon  problem 
with  deterministic  production  output.  Other  results  are  less  intuitive:  Let  the  financial 
holding  cost  at  the  disassembly  process  be  relatively  low,  and  the  physical  holding  cost  be 
the  same  in  both  processes.  In  this  case,  equation  14  may  be  approximated  by  the 


expression 


Equation  15. 


where  h/is  the  financial  holding  cost,  incurred  at  the  renovation  process  only,  and  hph  is  the 
physical  holding  cost,  of  the  same  magnitude  in  both  processes.  Hence, 

■  The  number  of  renovation  cycles  increases  with  the  financial  cost  of  the 
remanufacturing  operation.  This  happens  because  by  increasing  the  number  of 
cycles,  the  size  of  finished  goods  inventory  reduces,  which  drives  the  financial 
holding  cost. 

■  The  number  of  renovation  cycles  decreases  with  the  physical  handling  cost.  This  is 
an  indirect  effect.  Increasing  the  number  of  renovation  cycles  implicitly  reduces  the 
lot-size  in  the  disassembly  process,  hence,  reducing  the  physical  holding  cost  at  this 
level. 

■  If  the  expected  renovation  yield  is  low,  and  the  physical  holding  cost  is  relatively 
high,  there  will  be  as  many  renovation  cycles  as  disassembly  cycles.  This  happens  if 
the  expression  inside  the  square  root  is  less  than  1  (or  even  negative),  implying  that 
n*=  1. 

The  last  effect  clarifies  why,  in  some  remanufacturing  operations,  the  manager 
chooses  not  to  hold  inventory  between  the  two  events.  In  these  environments,  once  the  lot 
of  used  goods  is  disassembled,  it  proceeds  immediately  to  the  renovation  area.  This 
behavior  is  justifiable  because  handling  an  excessive  stock  of  disassembled  goods  may  be 
quite  problematic  if  storage  space  is  at  a  premium.  However,  if  physical  handling  is  not 
costly,  it  is  likely  that  the  renovation  station  will  process  smaller  lots  than  the  disassembly 
station. 


Example 


A  remanufacturing  facility  faces  an  annual  demand  of  600  units  of  a  certain  electric 
motor  series.  The  facility  has  access  to  an  ample  supply  of  used  motors  to  repair  at  a  small 
cost.  Holding  costs  have  been  estimated  as  hf^d  =  0.5,  hf_r  =  4,  hph_d  =  hp^j  =  2.  Moreover, 
ordering  and  setup  costs  have  been  estimated  as  kd  =  30  and  kr  =  6.  Pre-inspection  yield  for 
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each  lot  is  uniformly  distributed  between  0.5  and  0.95.  Final  inspection  yield  is  also 
uniformly  distributed — between  0.75  and  0.95.  Under  these  conditions,  we  find  that  n/o  =  2 
and  Phi  =  3.  Since  X(3)  =  61 .9  <  63.6  =  X(2),  we  conclude  that  n*  =  3.  Hence,  K{3)  =  80.9 
and  H{3)  =  2.84,  leading  to  Q(3)  =  185;  and  expected  inventory  management  cost  is 
minimized  at  C(3)  =  525.  The  following  graph  shows  the  expected  inventory  costs  at 
different  (Q,  n)  combinations. 

Figure  3  illustrates  that  the  operating  cost  does  not  change  significantly  close  to  the 
optimal  value  solution  (185,  3).  The  cost  increase  for  erring  in  just  one  dimension  (either  lot 
size  or  number  of  renovation  cycles)  is  quite  minor,  but  simultaneous  errors  in  both 
dimensions  can  easily  increase  operating  costs  by  50%  or  more.  Consequently,  the 
remanufacturing  facility  must  be  careful  deciding  the  inventory  policy  associated  with  its 
production  process  to  ensure  that  the  operating  cost  is  remains  close  to  its  theoretical 
optimum. 


□  750-800 

■  700-750 

■  650-700 

■  600-650 

■  550-600 

□  500-550 

■  450-500 

□  400-450 


n 


Figure  3.  Expected  Inventory  Costs  at  Different  (Q,  n)  Combinations 

Conclusion 

We  have  proposed  an  inventory  policy  for  multi-echelon  remanufacturing  operations 
in  which  the  first  echelon  corresponds  to  the  product  disassembly  and  sorting  operations, 
and  the  second  echelon  corresponds  to  the  repair,  renovation  and  final  inspection 
operations.  The  separation  between  these  two  sets  of  operations  is  important  because  they 
present  sizable  yield,  affecting  the  holding  cost  at  each  level  of  the  process.  We  find  a 
simple  inventory  policy  built  upon  the  familiar  structure  of  the  economic  order  quantity, 
leading  to  the  optimal  disassembly  lot  size  and  the  number  of  renovation  cycles  per 
disassembly  event. 
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This  policy  is  useful  in  DoD  depots,  where  large  remanufacturing  programs  are 
engaged  periodically  for  the  recovery  of  valuable  durable  assets.  We  intend  to  extend  this 
study  by  testing  the  policy  provided  herein  in  actual  remanufacturing  operations  in  the  DoD. 
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Abstract 

A  central  vision  of  B2B  e-commerce  is  that  of  an  electronic  marketplace  that  would  bring 
suppliers  with  major  buyers  of  goods  and  services  for  the  purpose  of  conducting  “frictionless” 
commerce.  The  hope  is  that  these  suppliers  would  compete  on  price,  transactions  would  be 
automated  and  low  cost,  and  as  a  result,  the  price  of  goods  and  services  would  fall.  Numerous 
Internet  marketplaces  came  into  being  during  the  Internet  boom;  however,  an  almost  equal 
number  disappeared  following  the  Internet  bubble  burst.  Still,  many  survive  today  based  on  a 
variety  of  models  that  are  quite  successful.  If  a  right  model  is  selected,  it  could  help  large 
organizations,  like  the  DoD,  achieve  great  efficiencies  for  their  acquisition  and  procurement 
processes. 

The  objective  of  the  paper  is  to  examine  models  for  classifying  and  differentiating  the 
business  functionality  provided  by  Internet  marketplaces  and  to  investigate  the  impact  of  the 
various  models  on  government  and  DoD  acquisition.  The  models  will  consider  such  variables  as 
types  of  goods  and  services  purchased,  how  these  goods  and  services  are  purchased,  pricing 
mechanisms,  the  characteristics  of  the  markets,  and  ownership  of  marketplace. 

Introduction 

Business-to-business  (B2B)  e-commerce  refers  to  transactions  between  businesses 
conducted  electronically  over  the  Internet,  intranets,  extranets,  or  private  networks.  Such 
transactions  may  be  conducted  between  a  business  and  its  suppliers  or  between  a  business 
and  any  other  business. 

It  is  estimated  that  in  2003,  B2B  e-commerce  in  the  United  States  was  a  $1 .5  trillion 
business.  This  represents  about  11%  of  the  total  B2B  trade  estimated  at  $13.5  trillion  (Laudon  & 
Traver,  2004).  Gartner  group  predicts  this  percentage  to  grow  steadily  to  reach  over  40%  in 
2010.  Forrester’s  research  predicts  a  higher  percentage  of  53%. 
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There  are  many  potential  benefits  of  B2B  e-commerce.  These  benefits  depend  on  the 
model  used,  but  are  thought  to  include  the  following: 

■  Significant  cuts  in  acquisition  cost 

■  Expediting  cycle-time 

■  Reducing  errors  and  improving  quality  of  service 

■  Seamless  integration  with  suppliers 

■  Ability  to  have  purchasing  data  instantly 

■  Reducing  inventory  levels  and  costs 

■  Immediate  response  to  changes  in  customer  purchasing  patterns 

■  Facilitating  mass  customization 

■  Increasing  opportunities  of  collaboration  between  buyers  and  sellers 

In  this  paper,  we  examine  models  for  classifying  and  differentiating  the  business 
functionality  provided  by  B2B  e-commerce  and  examine  the  impact  of  the  various  models  on 
government  and  DoD  acquisition.  The  models  will  consider  such  variables  as  types  of  goods 
and  services  purchased,  how  these  goods  and  services  are  purchased,  pricing  mechanisms, 
the  characteristics  of  the  markets,  and  ownership  of  marketplace. 

B2B  Characteristics 

There  are  many  ways  to  characterize  B2B  transactions.  In  this  paper,  we  differentiate 
between  different  types  of  B2B  transactions  based  on  the  following  characteristics:  Parties  to 
the  transaction,  types  of  transactions,  types  of  products  and  services  procured,  the  direction  of 
trade,  and  number  and  form  of  participation. 

Parties  to  the  Transaction 

B2B  commerce  can  be  conducted  directly  between  a  buyer  and  seller  or  through  a  third- 
party  intermediary. 

Types  of  Transactions 

There  are  two  types  of  transactions:  Spot  purchases  and  long-term  sourcing.  Spot 
purchases  refer  to  the  purchasing  of  goods  and  services  as  they  are  needed  at  the  prevailing 
market  prices.  Long-term  sourcing  refers  to  purchases  made  through  long-term  contracting 
agreements  that  are  negotiated  between  the  buyers  and  the  sellers. 

Types  of  Products  and  Services 

There  are  two  types  of  Products  and  Services:  Direct  and  Indirect.  Direct  products  and 
services  are  used  directly  in  making  the  product,  such  as  wood  in  furniture  or  paper  in  a  book. 
Direct  products  and  services  are  usually  purchased  in  large  quantities  using  long-term  sourcing. 
Indirect  products  and  services  (such  computer  equipments,  lights,  or  tools)  support  production, 
but  are  not  directly  involved  in  creating  the  end  product.  They  are  usually  referred  to  as 
maintenance,  repairs,  and  operations  (MROs). 

Direction  of  Transactions 
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B2B  transactions  can  be  classified  as  either  vertical  or  horizontal.  A  vertical  market  is 
one  that  provides  products  and  services  for  a  specific  industry.  Examples  include  cars,  steel,  or 
electronics.  Horizontal  markets  refer  to  markets  that  serve  many  different  industries.  Examples 
are  office  supplies,  computers,  and  tools. 

Number  and  Form  of  Participation 

There  are  four  types  of  electronic  marketplace  participation:  1)  Sell-side,  2)  Buy-side,  3) 
Exchanges,  and  4)  Collaborative  commerce. 

In  sell-side  commerce,  there  is  one  seller  that  does  all  the  selling  to  many  buyers.  In 
buy-side  commerce,  there  is  one  buyer  that  does  all  the  buying  from  many  sellers.  Both  types 
are  collectively  referred  to  as  company-centric  electronic  commerce,  because  they  address  a 
single  company  buying  or  selling  needs. 

Exchanges  are  many-to-many  electronic  marketplaces,  where  many  buyers  and  many 
sellers  meet  in  electronic  markets  to  conduct  business  transactions.  Exchanges  are  usually 
owned  and  managed  by  a  third  party  or  by  a  consortium,  and  are  open  to  all  interested  parties, 
and  are,  thus,  considered  public  electronic  marketplaces. 

Collaborative  commerce  goes  beyond  selling  and  buying  activities  and  includes  activities 
that  represent  more  than  financial  transactions — such  as  communication  and  sharing  of 
information,  planning,  design,  manufacturing,  and  management.  Collaborative  commerce  is 
relationship-based  rather  than  transactions-based  and  bears  resemblance  to  internal  workgroup 
collaborative  environments. 

Buy-side  Electronic  Marketplaces  Models 

Under  these  models,  a  buyer  opens  an  electronic  marketplace  on  its  own  servers  and 
invites  potential  suppliers  to  bid  on  the  products  and  services  that  the  buyer  needs.  This 
invitation  could  take  the  form  of:  1)  a  request  for  Quote  (RFQ),  or  2)  an  invitation  for  a  reverse 
auction.  An  example  of  the  former  is  FedBizOpps  (2007)  and  GSA  e-buy  (2007).  An  example  of 
the  latter  is  NAVSUP  NavyAuctions  (2007). 

FedBizOpps  is  the  single  government  point-of-entry  (GPE)  for  Federal  government 
procurement  opportunities  over  $25,000.  Government  buyers  are  able  to  publicize  their 
business  opportunities  by  posting  information  directly  to  FedBizOpps  via  the  Internet.  Using  the 
same  portal,  commercial  vendors  seeking  Federal  markets  for  their  products  and  services  can 
search,  monitor  and  retrieve  opportunities  solicited  by  the  entire  Federal  contracting  community. 

E-buy  is  an  electronic  Request  for  Quote  (RFQ)/Request  for  Proposal  (RFP)  system 
designed  to  allow  Federal  buyers  to  request  information,  find  sources,  and  prepare  RFQs/RFPs, 
online,  for  millions  of  services  and  products  offered  through  GSA's  Multiple  Award  Schedule 
(MAS)  and  Government-wide  Acquisition  Contracts  (GWAC). 

Navy  Auctions  is  a  secured  Internet  portal  that  allows  online  suppliers  to  compete  in 
real-time  for  contracts  by  lowering  their  prices  as  they  see  other  offers.  In  its  first  reverse 
auction,  the  Navy  estimates  that  they  achieved  savings  of  28.9%  over  the  historical  price  for 
these  items.  The  auction  lasted  51  minutes,  and  the  contract,  valued  at  $2,375  million,  was 
awarded  within  an  hour  of  the  reverse  auction  closing. 
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B2B  Exchanges 


As  discussed  earlier,  exchanges  are  electronic  marketplaces  where  many  buyers  and 
sellers  meet  to  buy  and  sell  goods  and  services.  Exchanges  are  known  under  different  names: 
e-marketplaces,  e-markets,  Internet  exchanges.  Net  marketplaces,  and  B2B  portals. 

Classification  of  Exchanges 

There  are  numerous  ways  of  classifying  exchanges.  We  use  an  approach  similar  to  that 
suggested  by  Kaplan  and  Sawhney  (2000)  and  Kerrigan,  Roegner,  Swinford  and  Zawada 
(2001).  The  classification  model  consists  of  a  2x2  matrix,  as  shown  in  Figure  1.  The  x-axis 
represents  the  types  of  goods  and  services  purchased  (indirect  goods  vs.  direct  goods),  and  the 
y-axis  represents  how  these  goods  and  services  are  purchased  (spot  purchases  vs.  long-term 
contractual  agreement).  The  intersection  of  these  dimensions  produces  four  cells  representing 
four  types  of  exchanges:  Horizontal  exchanges  (also  known  as  e-distributors),  horizontal 
distributors  (also  known  as  e-procurement),  vertical  exchanges  (also  known  as  independent 
exchanges),  and  vertical  distributors  (also  known  as  industry  consortia).  Each  of  these 
exchanges  seeks  to  provide  value  to  customers  in  different  ways.  We  discuss  each  type  of 
exchange  in  more  detail  in  the  following  sections. 
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Figure  1.  Types  of  Internet  Marketplaces 

Horizontal  Exchanges 

Horizontal  exchanges  are  independently  owned  intermediaries  that  offer  individual 
customers  a  single  source  from  which  to  make  spot  purchases  of  indirect  or  MRO  goods.  They 
operate  in  a  horizontal  market  that  serves  many  different  industries  with  products  from  many 
different  suppliers.  Horizontal  exchanges  are  usually  “public”  markets  that  any  firm  can 
participate  in.  They  usually  charge  fixed  prices,  and  their  owners  make  money  by  charging  a 
markup  on  products  they  distribute.  The  primary  benefits  to  customers  are  lower  search  costs, 
lower  transaction  costs,  wide  selection,  rapid  delivery,  and  low  prices.  An  example  of  a 
horizontal  exchange  is  Grainger  (2007)  and  the  DoD  Email  (2007). 

The  DoD  Email  was  launched  by  the  Defense  Logistics  Agency  (DLA)  in  1998  as  the 
DLA  Email.  It  was  created  to  leverage  purchasing  power  across  agencies  to  provide  the  Military 
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Services  and  other  Federal  Government  Agencies  with  volume  discounts  from  Military  and 
Commercial  suppliers.  Its  mission  is  indicated  in  the  FY99  DoD  Authorization  Act  \Nh\ch  states, 
“the  Joint  Electronic  Commerce  Program  Office  of  the  Department  of  Defense  shall  develop  a 
single,  defense-wide  electronic  mall  system,  which  shall  provide  a  single,  defense-wide 
electronic  point  of  entry  and  a  single  view,  access,  and  ordering  capability  for  all  Department  of 
Defense  electronic  catalogs.”  DLA  was  named  the  executive  agent  for  the  DoD  Email,  which 
remains  dedicated  to  its  DoD-wide  mission. 

There  are  currently  over  28,000  user  accounts  on  the  DoD  Email  with  500  new  users 
added  each  week.  These  users  represent  the  DoD  (All  Services,  National  Guard,  Reserves)  as 
well  as  other  Federal  Agencies  (DHS,  FBI,  etc.).  More  than  850  Commercial  Contracts  are 
currently  hosted  on  DoD  Email,  with  additional  catalogs  added  weekly.  The  DoD  Email  has 
shown  tremendous  growth — with  a  sales  increase  from  $14M  in  FY02  to  $336M  through  April  of 
FY05. 

Horizontal  Distributors 

Similar  to  horizontal  exchanges,  horizontal  distributors  are  independently  owned 
intermediaries  connecting  hundreds  of  online  suppliers  offering  millions  of  MRO  goods  to 
thousands  of  business  firms.  They  differ  from  horizontal  exchanges  in  that  they  operate  in  a 
horizontal  market  in  which  long-term  contractual  purchasing  agreements  are  used  to  buy 
indirect  goods.  Another  important  difference  is  that  horizontal  distributors  usually  provide  value- 
chain  management  services,  which  could  include  the  automation  of  a  firm’s  entire  procurement 
process  on  the  buyer  side  and  the  automation  of  the  selling  business  processes  on  the  seller 
side.  For  buyers,  this  includes  the  automation  of  purchase  orders,  requisitions,  invoicing,  and 
payments.  For  suppliers,  it  includes  the  automation  of  catalog  creation,  content  management, 
order  management,  order  fulfillment,  invoicing  and  shipment.  Horizontal  distributors  make 
money  by  charging  a  percentage  of  each  transaction,  licensing  consulting  services  and 
software,  and  assessing  network  use  fees. 

The  two  largest  horizontal  distributor  players  are  Ariba  (2007)  and  Perfect  Commerce, 
previously  CommerceOne  (2007).  Although  some  Government  and  DoD  initiatives  include  some 
characteristics  of  this  model  (e.g.,  e-buy),  there  is  no  Government  or  DoD  effort  that  provides  a 
full  automation  of  the  acquisition  process  on  the  buyer  side  and  the  automation  of  the  selling 
process  on  the  seller  side. 

Vertical  Exchanges 

Vertical  exchanges  are  independently  owned  online  marketplaces  that  connect  hundreds 
of  suppliers  to  potentially  thousands  of  buyers  in  a  dynamic  real-time  environment.  They  are 
typically  vertical  markets  in  which  spot  purchases  can  be  made  for  direct  inputs  (both  goods  and 
services).  Similar  to  horizontal  exchanges,  the  benefits  for  buyers  include  reduced  search  costs 
and  lower  prices,  while  the  benefits  for  sellers  include  access  to  the  global  purchasing 
environment  and  opportunity  to  unload  production  overruns.  Vertical  exchanges  make  money  by 
charging  a  commission  on  each  transaction;  pricing  can  be  through  an  online  negotiation, 
auction,  RFQ,  or  fixed  prices.  Vertical  exchanges  are  “public”  markets  and  are  biased  in  favor  of 
the  buyer.  An  important  measure  of  success  for  vertical  exchanges  is  their  liquidity — which  is  a 
measured  by  the  numbers  of  buyers  and  sellers  in  the  market,  the  volume  of  transactions,  and 
the  size  of  transactions.  If  there  is  a  small  number  of  participants,  a  low  volume  of  small 
transactions,  an  exchange  usually  fails. 
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Examples  of  Vertical  Exchanges  include  E-steel  (2006),  a  spot  market  for  steel  products 
and  Foodtrader  (2003),  a  spot  market  for  the  food-products  industry. 

Vertical  Distributors 

Vertical  distributors,  also  known  as  industry  consortia,  are  industry-owned  vertical 
markets  in  which  long-term  contractual  purchases  of  direct  inputs  can  be  made  from  a  limited 
set  of  invited  participants.  They  serve  to  reduce  supply-chain  inefficiencies  by  unifying  the 
supply  chain  for  an  industry  through  a  common  network  and  computing  platform.  They  make 
money  through:  1 )  Industry  members  who  pay  for  the  creation  of  the  site  and  contribute  initial 
operating  capital  and  2)  Buyer  firms  who  pay  transaction  and  subscription  fees.  The  pricing 
mechanism  of  this  model  ranges  from  auctions  to  fixed  prices  to  RFQs.  The  bias  of  industry 
consortia  is  toward  large  buyers  who  benefit  from  competitive  pricing.  Benefit  to  suppliers  is 
from  access  to  large-buyer-firm  procurement  systems,  long-term  stable  relationships,  and  large- 
order  sizes. 

There  are  numerous  vertical  distributors  in  many  industries,  with  many  industries  having 
more  than  one.  The  industries  with  the  most  common  consortia  are  metals,  chemicals,  and 
retail.  The  long-term  viability  of  Vertical  Distributors  is  yet  to  be  seen. 

Examples  of  Vertical  Distributors  are  Covisint  (2007)  for  the  automotive  industry  and 
Exostar  (2007)  for  Defense  and  Aerospace. 

How  Are  Exchanges  Evolving? 

Exchanges’  capabilities  are  evolving  rapidly  and  growing  increasingly  sophisticated. 
Figure  2  depicts  some  of  these  changes.  Horizontal  exchanges  are  moving  away  from  being 
simple  electronic  marketplaces  toward  more  active  and  sustained  relationships  with  buyer 
companies  by  providing  added-value  services  and  participating  in  industry  consortia  as 
suppliers  of  indirect  goods.  These  added-value  services  include  the  automation  of  part  or  the 
entire  procurement  process  on  the  buyer  side  and  the  automation  of  the  selling  business 
processes  on  the  seller  side.  For  example,  selling  value-added  services  could  include  Web 
store  fronts,  the  ability  to  configure  and  price  products,  and  customer  support,  such  as  order- 
status  monitoring,  demand  planning  and  collaboration. 

Similarly,  vertical  exchanges  are  being  absorbed  into  industry  consortia  as  many  were 
not  attracting  enough  players  to  achieve  liquidity.  Another  important  trend  in  exchanges  is  the 
movement  from  simple  transactions  of  spot  purchases  to  longer-term  sourcing  agreements 
involving  both  direct  and  indirect  goods  (Wise  &  Morrison,  2000). 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  -  351  - 


What  Organizations  Buy 

Indirect  Goods  and  Direct  Goods  and 
Services  (MROs)  Services 


How 

Organizations 

Buy 


Spot 

Purchasing 


Long-term 

Sourcing 


Horizontal 

Markets 


Horiz 

Exchc 

(E-Distri 

ontal 

inges 

it^ors) 

Ven 

Exchi 

(In^ 

E^h' 

tical 

anges 

endent 

nges) 

Hori: 

Distri 

(E-Proc^ 

)ntal  // 
utojTY 

^  Vei 

x^istri 

xi 

ical 

utors 

>try 

prtia) 

<= 


Vertical 

Markets 


Figure  2.  Evolution  of  Exchanges 

Collaborative  Commerce 

Collaborative  commerce  is  used  to  describe  Web-based  communication  environments 
that  extend  beyond  procurement  to  include  coordinating  trans-organizational  business 
processes.  Collaborative  Commerce  permits  buyer  firms  and  principle  suppliers  to  share 
product  design  and  development,  marketing,  inventory,  production  scheduling,  and  unstructured 
communications.  It  generally  starts  as  an  enterprise  resource  planning  (ERP)  system  in  a  single 
firm  that  is  then  expanded  to  include  the  firm’s  major  suppliers.  This  fact  differentiates  private 
industrial  networks  from  consortia,  which  are  usually  owned  collectively  by  major  firms  through 
equity  participation.  Collaborative  commerce  is  considered  a  buyer-side  solution  with  buyer 
biases.  It  is  the  most  prevalent  form  of  Internet-based  B2B. 

A  good  example  of  the  benefits  of  collaborative  commerce  is  its  collaborative  resource 
planning,  forecasting,  and  replenishment  (CPFR) — which  require  the  collaborating  members  to 
forecast  demand,  develop  production  scheduling  plans,  coordinate  shipping,  warehousing,  and 
replenishment  activities  to  ensure  retail  and  warehouse  shelf  spaces  are  replenished  “just  in 
time.”  This  approach  could  potentially  realize  hundreds  of  millions  of  dollars  in  excess  inventory 
and  production  savings  and,  therefore,  produce  large  benefits  to  justify  the  cost  of  developing 
the  collaboration  network. 

A  second  example  of  collaborative  commerce  is  demand-chain  visibility,  in  which  excess 
capacity  and  supplies  in  the  supply-and-distribution  chain  is  visible  to  all  members  of  the  chain. 
Adjustments  could  then  be  made  in  real-time  to  production  capacities  to  avoid  excess 
inventories  that  usually  create  pressure  to  discount  merchandise,  reducing  profits  to  all  parties 
involved. 

Collaborative  commerce  faces  many  implementation  barriers.  First,  participating  firms 
are  required  to  share  sensitive  data  with  their  business  partners.  This  is  a  particularly  major 
impediment  for  government  and  DoD  organizations.  Second,  integrating  collaborative  networks 
into  existing  ERP  systems  and  EDI  networks  is  expensive  and  time-consuming.  Third, 
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collaborative  commerce  requires  a  change  in  mind-set  and  behavior  of  employees,  which 
constitutes  a  major  paradigm  shift. 

Conclusions 

While  B2B  e-commerce  today  accounts  for  a  small  percentage  of  the  total  B2B,  it  is 
growing  steadily  and  expected  to  reach  40%  -  50%  of  the  total  B2B  trade  in  a  few  years.  B2B 
transactions  promise  to  help  organizations  run  more  efficiently  by  achieving  significant  cost 
savings  and  reductions  in  cycle-time. 

Many  models  of  B2B  e-commerce  have  emerged,  each  providing  different  functionality 
for  the  business  it  supports.  Initially,  B2B  models’  focus  was  commerce  and  transaction 
execution.  However,  newer  models’  focus  is  increasingly  on  value-added  services  and  support 
for  cross-enterprise  collaboration.  It  is  important  for  the  DoD  to  examine  these  models,  their 
characteristics,  and  trends  in  order  to  leverage  the  future  of  B2B  and,  therefore,  to  do  business 
more  efficiently. 
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